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CMM  Appraisal  Framework 


Abstract:  This  technical  report  describes  version  1.0  of  the  CMM  Appraisal  Framework 
(CAF).  This  framework  describes  the  common  requirements  used  by  the  CMM-Based  Ap¬ 
praisal  (CBA)  project  in  developing  appraisal  methods  based  on  the  Capability  Maturity  Model 
(CMM)  for  Software,  Version  1.1  [Paulk  93a].  The  CAF  provides  a  framework  for  rating  the 
process  maturity  of  an  organization  against  the  CMM.  The  CAF  includes  a  generic  appraisal 
architecture  for  CMM-based  appraisal  methods  and  defines  the  requirements  for  developing 
CAF  compliant  appraisal  methods. 

1  Background  and  Context 

1 .1  Who  is  the  Intended  Audience  for  this  Document? 

Anyone  interested  in  the  conduct  of  process  appraisals  can  benefit  from  reading  this  docu¬ 
ment.  The  primary  targets  for  this  document  are: 

•  Lead  appraisers. 

•  Appraisal  method  developers. 

•  Members  of  appraisal  teams. 

•  Sponsors  of  appraisals. 

•  Process  improvement  practitioners. 

1 .2  How  is  this  Document  Structured  and  What  Is  Its  Scope? 

Part  1  of  the  CAF  includes  background  information  relating  the  CAF  to  key  user  consider¬ 
ations.  It  addresses: 

•  What  the  CAF  is. 

•  How  the  CAF  can  be  used. 

•  How  the  CAF  relates  to  existing  reference  models  and  appraisal  methods. 

•  How  the  CAF  relates  to  existing  documentation. 

•  The  sponsor  needs  which  drove  CAF  development. 

•  The  specific  objectives  of  the  CAF. 

•  The  principles  and  design  constraints  that  guided  CAF  development. 

Part  2  describes  the  architecture  of  CAF  appraisal  methods.  It  describes  the  context  of  CAF 
appraisals,  identifies  the  major  CAF  appraisal  method  families,  itemizes  the  activities  that 
make  up  CAF  appraisals,  and  describes  the  major  data  transformations  that  occur  during  CAF 
appraisal  execution. 

Part  3  provides  the  CAF  appraisal  method  requirements.  It  is  subdivided  into  sections  that: 
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•  Group  requirements  by  topic. 

•  Provide  related  appraisal  activity  data  flow  diagrams. 

•  Provide  explanations  of  requirements  that  include  definitions,  rationale, 
factors  for  consideration,  and  examples. 

Appendices  provide  the  following  information: 

•  Appendix  A  provides  a  list  of  all  CAP  requirements. 

•  Appendix  B  shows  a  mapping  of  the  CMM  key  practices  to  KPA  goals. 

•  Appendix  C  is  a  glossary  of  key  terms. 

•  Appendix  D  provides  a  list  of  references. 

The  scope  of  the  CAP  V1 .0  pertains  to  appraisal  methods  that  are  SEI  CMM-based.  The  CAP 
is  not  an  appraisal  method.  Rather,  it  is  a  set  of  minimum  requirements  and  guidelines  to  be 
used  by  method  developers  in  designing  methods  that  are  CAP  compliant.  Just  as  the  CMM 
is  an  abstract  model  that  needs  to  be  instantiated  in  real  practice  in  a  software  organization  to 
have  meaning,  the  CAP  needs  to  be  instantiated  in  a  real  appraisal  method. 

All  CAP  requirements  are  listed  in  Section  3  of  this  document.  The  reader  should  not  construe 
any  discussion  in  Sections  1  and  2  to  be  implicit  requirements.  These  two  parts  contain  ex¬ 
planatory  information  which  will  help  to  prepare  the  reader  for  the  requirements  section. 

1 .3  How  Can  This  Document  Be  Used? 

This  document  can  be  used  in  several  ways,  depending  on  end  user  perspective. 

The  CAP  provides  lead  appraisers  with  an  aid  for  training  appraisal  team  members. 

The  CAP  provides  appraisal  method  developers  with  a  tool  for  designing  and  developing 
CMM-based  appraisal  methods. 

The  CAP  provides  appraisal  team  members  with  an  explanation  of  appraisal  requirements. 
The  CAP  supports  appraisal  sponsors  and  lead  appraisers  in: 

•  Assessing  whether  or  not  a  specific  appraisal  method  satisfies  sponsor 
business  needs. 

•  Assessing  the  trade-offs  of  using  a  specific  appraisal  method. 

The  CAP  supports  process  improvement  practitioners  by  letting  them  know  how  an  appraisal 
team  will  base  rating  judgments  for  individual  CMM  components  in  a  CAP  compliant  appraisal 
method. 
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1.4  What  is  the  CMM  Appraisal  Framework  (CAF)? 

The  CAF  is  a  framework  for  developing,  defining,  and  using  appraisal  methods  based  on  the 
Software  Engineering  Institute’s  (SEI)  Capability  Maturity  Model  (CMM)  for  Software.  The 
CAF  provides  a  framework  for  rating  the  process  maturity  of  an  organization  against  a  gener¬ 
ally  accepted  reference  model  through  the  use  of  an  appraisal  method.  The  CAF  V1.0  uses 
the  CMM  Version  1.1  [Paulk  93a,  Paulk  93b]  as  its  associated  reference  model.  The  CAF  and 
its  revisions  will  be  applicable  to  other  models  in  the  future  provided  they  comply  with  the  CMM 
architecture.  If  future  versions  of  the  CMM  have  an  architecture  different  from  CMM  Version 
1.1  so  that  portions  of  the  existing  CAF  are  no  longer  applicable,  an  updated  version  of  the 
CAF  will  be  written. 

The  CAF  includes  an  architecture  for  a  generic  CAF  appraisal  method  and  a  description  of 
CAF  appraisal  method  requirements.  How  method  developers  implement  CAF  requirements 
is  a  method  specific  choice.  The  CAF  does  not  directly  measure  risk  or  process  improvements 
or  identify  lost  market  opportunities.  It  is  up  to  users  of  appraisal  method  outcomes  to  translate 
the  results  of  CAF  appraisals  into  meaningful  information  relative  to  the  sponsor’s  business 
needs. 

When  an  appraisal  method  meets  all  of  the  CAF  requirements,  it  is  said  to  be  fully  CAF  com¬ 
pliant.  Future  versions  of  the  CBA  project’s  SCE  and  CBA  IPI  appraisal  methods  will  be  CAF 
compliant. 

Requests  can  be  made  to  the  SEI  to  determine  whether  or  not  an  appraisal  method  developed 
outside  of  the  SEI  is  CAF  compliant.  The  SEI  will  charge  a  fee  for  such  services.  The  point  of 
contact  for  such  a  request  is  SEI  customer  relations. 

The  CAF  was  designed  to  help  improve  consistency  within  a  specific  appraisal  method  and 
across  different  appraisal  methods  and  to  help  appraisal  method  developers,  sponsors,  and 
users  understand  trade-offs  associated  with  various  methods.  Not  all  appraisal  methods  are 
expected  to  be  fully  CAF  compliant.  Rather,  the  CAF  provides  a  standard  by  which  one  can 
evaluate  CAF  compliance  and  assess  the  trade-offs  associated  with  a  method  relative  to 
meeting  specific  sponsor  business  and  appraisal  goals. 

Figure  1-1  shows  the  relationships  among  reference  models,  the  CAF,  and  appraisal  meth¬ 
ods. 
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Figure  1-1 :  Relationships  Among  the  CMM,  the  CAF,  and  CAF  Compliant  Appraisal 

Methods 

The  CMM  provides  a  framework  forjudging  the  maturity  of  an  organization’s  software  process. 
The  CAF  provides  requirements  and  guidelines  for  developing,  defining,  and  using  appraisal 
methods  based  on  the  CMM.  Together  the  CMM  and  the  CAF  describe  “what”  must  be  accom¬ 
plished  by  CAF  compliant  appraisal  methods.  The  appraisal  methods  themselves  detail  “how” 
to  transform  an  organization’s  software  process  data  into  information  of  value  to  meeting  an 
organization’s  business  needs. 


1 .5  What  is  the  Relationship  of  the  CAF  to  Other  CBA  Documents? 

In  the  CAF  VI  .0,  the  CMM  VI  .1  is  the  reference  model  used.  Multiple  appraisal  methods  can 
be  developed  that  are  consistent  with  CAF  requirements.  Each  appraisal  method  will  have  its 
own  documentation.  A  typical  appraisal  method  document  suite  will  address  such  issues  as: 

•  Use  by  lead  appraisers  and  appraisal  sponsors  to  determine  the  applicability 
of  a  method  to  a  particular  business  situation. 

•  Use  by  appraisal  sponsors  to  integrate  appraisals  into  their  business 
operations  (for  example,  to  identify  the  manner  in  which  an  appraisal  might 
be  integrated  into  the  software  supplier  acquisition  process). 

•  Use  by  appraisal  teams  to  carry  out  appraisals  and  to  provide  detailed 
instructions  for  each  of  the  activities  necessary  to  carry  out  an  appraisal. 

•  Use  by  appraisal  instructors  to  train  appraisal  teams. 

Modular  parts  of  these  documents  are  referred  to  as  appraisal  method  assets.  Figure  1-2  de¬ 
picts  the  relationships  between  CMM-based  appraisal  method  assets.  The  inner  ring,  the 
CMM,  is  the  model  used  as  a  framework  for  evaluating  an  organization’s  software  process. 
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The  CAF  is  the  framework  for  executing  an  appraisal  using  this  model.  The  next  ring  contains 
common  assets  --  those  artifacts,  tools,  and  techniques  that  can  be  shared  across  specific 
methods.  The  outer  ring  depicts  CAF  compliant  appraisal  methods,  which  use  the  published 
versions  of  the  CMM  and  CAF. 


Figure  1-2:  CBA  Concept  Diagram 

This  CBA  concept  of  appraisal  assets  acknowledges  and  supports  three  fundamental  charac¬ 
teristics  of  the  current  appraisal  environment: 

•  There  are  different  sponsor  and  user  needs  that  must  be  met  by  appraisal 
methods. 

•  There  are  many  potential  appraisal  method  users  in  the  software  community 
that  need  to  be  supported. 

•  Although  there  are  many  possible  types  of  appraisals,  many  of  them  share 
similar  objectives  and  are  variations  on  a  common  theme. 
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1 .6  What  Sponsor  Needs  Are  Addressed  in  the  CAF? 

The  CAF  is  intended  to  help  address  sponsors’  fundamental  appraisal  needs.  For  example: 

•  Ensure  that  appraisal  results  contribute  directly  to  software  process 
improvement. 

•  Optimize  appraisal  value. 

•  Ensure  appraisal  reliability. 

•  Facilitate  “buy-in”  by  the  appraised  organization. 

These  four  goals  can  be  operationalized  to  measure  the  success  of  any  particular  appraisal 
method.  For  example,  “success”  might  be  measured  as  the  percentage  of  the  findings  of  the 
method  that  are  translated  into  specific  process  improvement  actions.  Appraisal  value  could 
be  measured  by  cost,  time,  or  sponsor  surveys,  and  appraisal  reliability  can  be  tied  to  mea¬ 
sures  of  the  repeatability  of  its  process  and  predictability  of  its  results.  “Buy-in”  could  be  mea¬ 
sured  by  a  survey  of  the  appraised  organization’s  opinion  of  the  appraisal  results. 

The  ultimate  value  of  the  CAF  is  in  its  ability  to  meet  these  four  goals. 

1 .7  What  are  the  Objectives  of  the  CAF? 

The  CAF  is  intended  to  meet  several  objectives: 

•  Define  requirements  for  CAF  compliant  methods. 

•  Provide  guidance  for  comparing  different  methods. 

•  Define  CAF  compliant  method  components  and  the  relationships  among 
them. 

•  Make  public  an  explicit  definition  of  the  CAF  compliant  method  rating  process 
and  its  prerequisites. 

Meeting  these  objectives  should  indirectly  help  achieve  the  following: 

•  Reduce  inconsistencies  in  appraisal  method  use. 

•  Increase  comparability  of  different  appraisal  methods  and  their  results. 

•  Increase  predictability  and  consistency  of  appraisal  results. 

•  Increase  specificity  of  actionable  appraisal  results. 

•  Allow  flexibility,  but  encourage  consistency,  in  appraisal  method 
development. 
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Community  requirements  collected  by  the  SEI  during  the  1992-1993  time  frame  can  be  sum¬ 
marized  in  the  following  broad  areas: 

1 .  Baseline  the  SEI  appraisal  methods. 

2.  Make  the  methods  public. 

3.  Incorporate  the  CMM  VI. 1  into  the  SEI  methods. 

4.  Align  assessment  and  evaluation  methods  with  each  other. 

5.  Define  a  process  for  evolving  and  transitioning  these  SEI  methods. 

The  CAP  principally  addresses  the  fourth  item,  aligning  methods  with  each  other.  Methods  that 
are  CAP  compliant  should  be  consistent  with  one  another.  It  is  also  a  mechanism  for  achieving 
the  transition  of  SEI  methods  to  the  software  community  (fifth  item).  A  long  term,  indirect  goal 
of  defining  the  CAP  is  to  encourage  development  of  reusable  appraisal  method  assets  for  use 
by  future  method  developers. 

1 .8  What  Design  Goals  Are  Built  into  the  CAF? 

Several  design  goals  were  used  to  guide  the  development  of  the  CAP: 

•  Define  a  common  CMM-based  appraisal  architecture. 

•  Define  a  common  set  of  CMM-based  appraisal  activities. 

•  Ensure  appraisal  activities  are  conducted  by  a  team. 

•  Define  a  common  set  of  CMM-based  appraisal  data  requirements. 

•  Define  a  common  CMM-based  appraisal  rating  process. 

•  Base  the  rating  process  on  the  full  structure  of  the  reference  model  (e.g. 

CMM  components  -  key  practices,  common  features,  goals,  and  KPAs). 

•  Base  rating  judgments  (of  the  reference  model  components)  on: 

•  observations  by  the  appraisal  team 

•  rules  for  confirming  observations  and  model  coverage  (e.g.,  determining 
sufficiency  for  rating) 

•  Ensure  judgment  processes  used  by  the  appraisal  team: 

•  look  at  work  actually  done 

•  simplify  the  effort  to  reach  decisions 

•  are  intellectually  manageable 

•  are  linked  explicitly  to  data  coverage 

•  Consider  a  method  fully  CAP  compliant  if  it  implements  all  CAP 
requirements. 
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1.9  What  Are  the  Basic  Principles  That  Guided  CAP  Development? 

The  CAF  embodies  several  basic  principles  that  are  important  to  appraisal  method  sponsors 
and  users: 

•  Appraisal  teams  must  ensure  that  data  collection  covers  the  scope  of  the 
appraisal. 

•  Data  collected  must  incorporate  what  is  heard  and  seen  by  the  appraisal 
team. 

•  Data  collected  must  be  organized  according  to  the  reference  model  used. 

•  Full  coverage  of  the  reference  model  components  specified  in  the  appraisal 
plan  must  be  verified  in  order  to  rate  those  components. 

•  Ratings  must  be  based  on  the  processes  actually  implemented  and 
practiced. 

•  If  an  organization  implements  the  practices  in  the  reference  model,  they  must 
be  rated  satisfactorily  in  those  applicable  components. 

•  An  organization  doesn’t  have  to  implement  practices  as  described  in  the 
reference  model  to  satisfy  the  intent  of  the  reference  model  --  alternates  are 
acceptable  provided  that  they  support  the  key  process  area  (KPA)  goals. 

•  Two  types  of  risks  should  be  considered  by  appraisal  method  users:  risk 
inherent  in  the  method  chosen  to  be  performed,  and  risks  based  on  the 
appropriate  execution  of  that  method. 

1.10  How  Does  the  CAF  Relate  to  Previously  Available  SEI  Appraisal 
Methods? 

The  CAF  postdates  the  introduction  of  some  existing  SEI  appraisal  methods,  such  as  the  SPA 
Method  taught  to  SEI-licensed  vendors  and  SCE  Method  described  in  Software  Capability 
Evaluation  Version  2.0  Method  Description  [CBA  Project  94].  This  does  not  imply  that  the  CAF 
and  existing  appraisal  methods  are  inconsistent,  but  it  does  mean  that  these  existing  appraisal 
methods  may  not  be  fully  CAF  compliant  and  are  not  necessarily  consistent  with  each  other. 

The  genesis  of  the  CAF  is  from  “A  Method  for  Assessing  the  Software  Engineering  Capability 
of  Contractors”  [Humphrey  87b].  Since  the  time  that  the  CMM  came  into  existence,  the  soft¬ 
ware  community  has  evolved  and  matured,  and  limitations  with  the  existing  appraisal  methods 
became  apparent.  Existing  methods  can  continue  to  be  used  as  defined;  however,  it  must  be 
understood  and  acknowledged  that  they  may  not  be  CAF  compliant,  that  they  may  have  limi¬ 
tations,  and  that  there  may  not  be  consistency  between  methods. 
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2  Appraisal  Architecture 


This  section  of  the  document  will  describe  the  architecture  of  CAF  compliant  appraisal  meth¬ 
ods.  It  will  describe  the  context  of  CAF  appraisals  and  identify  the  major  appraisal  activities 
and  data  transformations  required  of  CAF  compliant  appraisal  methods. 


2.1  Appraisal  Context 

Appraisals  themselves  need  to  be  considered  as  part  of  a  larger  system.  Appraisals  apply  di¬ 
rectly  to  a  process  improvement  system.  Figure  2-1  below  shows  the  SEI’s  IDEAL  approach 
to  integrated  software  process  improvement.  IDEAL  stands  for  the  five  phases  of  the  ap¬ 
proach:  Initiating,  Diagnosing,  Establishing,  Acting,  and  Leveraging. 


Appraisals  are  part  of  the  Diagnosing  phase  of  the  IDEAL  process  improvement  life  cycle. 
They  are  used  to  characterize  the  current  practices  of  an  organization.  They  should  be  con¬ 
ducted  only  when  there  is  clear  commitment  on  behalf  of  their  sponsor  to  use  their  results.  Re¬ 
sults  of  appraisals  are  used  to  develop  recommendations  for  furthering  process  improvement 
activities. 
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2.2  Appraisal  Families 

Appraisals  may  be  grouped  into  families  based  on  their  goals.  Three  primary  families  of  ap¬ 
praisals  are  anticipated  based  on  customer/supplier  relationships: 

•  Appraisals  whose  primary  purpose  is  to  provide  information  to  customers 
useful  in  selecting  software  suppliers. 

•  Appraisals  whose  primary  purpose  is  to  provide  information  to  guide 
supplier’s  internal  process  improvement  efforts. 

•  Appraisals  whose  primary  purpose  is  to  provide  information  for  joint 
customer/supplier  process  improvement  and/or  risk  management  efforts. 

2.3  Appraisal  Method  Activities 

Figure  2-2  illustrates  the  primary  activities  that  make  up  a  CAF  compliant  appraisal  method. 
These  activities  are  divided  into  three  (sometimes  overlapping)  phases: 

•  Plan  and  Prepare  for  Appraisal. 

•  Conduct  Appraisal. 

•  Report  Results. 

Planning  and  preparing  for  an  appraisal  includes  analyzing  the  appraisal  requirements  and 
developing  an  appraisal  plan  to  address  those  requirements  as  well  as  selecting  and  prepar¬ 
ing  both  the  appraisal  participants  and  the  appraisal  team. 

Conducting  an  appraisal  includes  collecting  and  recording  appraisal  data,  consolidating  it  into 
a  manageable  set  of  observations,  validating  those  observations,  and  making  rating  judg¬ 
ments  based  on  those  observations. 

Reporting  results  includes  documenting  and  presenting  appraisal  outputs. 
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Figure  2-2:  Appraisal  Activities  and  Section  Numbers  in  This  Document 


Section  3  details  the  CAF  requirements  for  providing  guidance  for  each  of  these  activities. 

2.4  Appraisal  Related  Data  Transformations 

Figure  2-3  depicts  the  basic  data  transformations  that  occur  during  a  CAF  compliant  appraisal 
as  raw  data  is  turned  into  reports  detailing  appraisal  outputs.  Major  transformations  include: 

•  Recording  data  in  the  form  of  individual  notes. 

•  Consolidating  these  notes  into  a  manageable  set  of  observations. 

•  Judging  as  a  team  whether  or  not  these  observations  constitute  a  valid  set  of 
findings. 

•  Making  rating  judgments  based  on  these  findings. 

•  Producing  appraisal  reports. 

CAF  requirements  for  addressing  each  of  these  transformations  are  provided  in  Section  3. 
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Figure  2-3:  Appraisal  Data  Transformations 
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3  CMM  Appraisal  Framework  Method  Requirements 


This  section  describes  the  requirements  of  CAF  compliant  methods.  The  following  subsec¬ 
tions  group  these  requirements  by  topic.  Each  subsection  consists  of  a  list  of  requirements  fol¬ 
lowed  by  a  discussion  that  addresses  the  rationale  for  the  requirements  and  factors  to 
consider  in  implementing  the  requirements.  Flow  diagrams  are  included  to  illustrate  these  re¬ 
quirements. 

Requirements  are  numbered  R1 ,  R2,  R3,  etc.  and  begin  with  the  phrase,  “A  CAF  compliant 
appraisal  method  shall....”  Requirements  fall  into  three  categories: 

•  Those  which  identify  an  asset  (such  as  guidance  for  a  particular  activity)  that 
must  be  provided  by  a  CAF  compliant  method,  but  do  not  constrain  its 
design, 

•  Those  which  identify  an  asset  that  must  be  provided  by  a  CAF  compliant 
method  and  whose  design  must  meet  certain  minimum  criteria, 

•  Those  which  identify  an  asset  that  must  be  provided  by  a  CAF  compliant 
method  and  define  it  explicitly. 

A  list  of  all  CAF  requirements  is  included  in  Appendix  A. 

3.1  Method  Documentation 

3.1.1  Documenting  CAF  Compliance 

Requirement:  R1 .  A  CAF  compliant  appraisal  method  shall  be  documented,  including  at  a 
minimum: 

•  Identifying  the  version  of  the  CMM  and  the  CAF  on  which  it  depends. 

•  Documenting  the  manner  in  which  it  has  implemented  appraisal  method 
activities,  artifacts  and  guidance  required  by  the  CAF. 

Discussion:  Appraisal  team  leaders  must  be  able  to  select  CAF  appraisal  methods  that  are 
compatible  with  the  goals  of  the  appraisals  they  conduct.  In  order  to  do  so  they  must  be  able 
to: 


•  Understand  CAF  compliant  appraisal  methods. 

•  Determine  their  consistency  with  published  versions  of  the  CMM. 

•  Evaluate  their  compliance  with  published  versions  of  the  CAF. 

Appraisal  team  leaders,  in  selecting  appraisal  methods  for  particular  appraisal  situations,  must 
understand  the  trade-offs  associated  with  the  method’s  specific  implementation  of  appraisal 
activities,  artifacts,  and  goals.  Different  appraisal  methods  will  achieve  different  balances 
among  appraisal  cost,  disruption  to  the  appraised  organization,  and  accuracy  of  appraisal  out¬ 
puts.  Appraisal  team  leaders  must  be  able  to  understand  the  impacts  of  appraisal  method 
choices  on  each  of  these  appraisal  elements. 
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3.2  Appraisal  Phases 

Requirement:  R2.  A  CAP  compliant  method  shall  provide  guidance  for  three  phases  of  ap¬ 
praisal  execution: 

•  Plan  and  Prepare  for  Appraisal. 

•  Conduct  Appraisal. 

•  Report  Results. 

Discussion:  The  activities  associated  with  any  appraisal  fall  into  three  (sometimes  overlap¬ 
ping)  phases:  planning/preparation,  conduct,  and  reporting  results.  Activities  performed  in 
each  phase  have  significant  impacts  on  appraisal  results.  Figure  3-1  illustrates  these  three 
phases  of  appraisal  execution.  The  following  paragraphs  describe  the  requirements  for  pro¬ 
viding  guidance  for  each  of  these  phases  in  detail. 
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3.3  Plan  and  Prepare  for  Appraisal 

Planning  and  preparation  are  the  key  to  success  of  any  appraisal.  As  illustrated  in  Figure  3-2, 
planning  and  preparation  involve  analyzing  the  appraisal’s  requirements,  selecting  and  pre¬ 
paring  the  appraisal  team,  selecting  and  preparing  the  appraisal  participants,  and  developing 
and  documenting  the  appraisal  plan.  The  following  subparagraphs  specify  CAF  requirements 
for  addressing  each  of  these  topics. 


Conduct  Appraisal 


Report  Results 

Figure  3-2:  Plan  and  Prepare  for  Appraisal 


3.3.1  Analyze  Requirements 

Analyzing  the  requirements  for  a  particular  appraisal  includes  evaluating  the  sponsor’s  ap¬ 
praisal  goals  and  constraints,  determining  that  these  goals  can  be  achieved  within  the  con¬ 
straints  of  a  specific  appraisal  method,  determining  the  appraisal  scope,  and  securing  the 
appraisal  sponsor’s  commitment  to  proceed  with  the  appraisal  process. 
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3.3.1. 1  Identify  and  Evaluate  Goals  and  Constraints 

Requirement:  R3.  A  CAP  compliant  appraisal  method  shall  provide  guidance  for: 

•  Identifying  an  appraisal’s  goals  and  constraints. 

•  Determining  its  suitability  for  the  appraisal  in  light  of  those  goals  and 
constraints. 

Discussion:  Different  types  of  appraisals  have  different  goals.  These  goals  vary  widely  and 
address  such  diverse  business  needs  as  selecting  software  suppliers  and  facilitating  internal 
process  improvement  efforts.  A  CAP  appraisal  method  may  be  more  suitable  for  performing 
some  types  of  appraisals  than  others.  Even  when  a  CAP  appraisal  method  is  designed  to  sup¬ 
port  a  specific  type  of  appraisal,  the  constraints  on  performing  a  particular  appraisal  of  that 
type,  such  as  availability  of  participants  for  interviews,  may  affect  the  appraisal  method’s  use¬ 
fulness.  Appraisal  sponsors  and  team  leaders  should  be  able  to  evaluate  the  trade-offs  of 
achieving  appraisal  goals  with  a  given  appraisal  method.  With  the  above  guidance,  this  eval¬ 
uation  can  occur  early  in  the  planning  process. 

3.3.1 .2  Establish  Scope 

The  scope  of  an  appraisal  includes  its: 

•  Capability  Maturity  Model  (CMM)  scope. 

•  Organizational  scope. 

3.3.1. 2.1  CMM  Scope 

Requirement:  R4.  A  CAP  compliant  appraisal  method  shall  provide  guidance  for  selecting  an 
appraisal’s  CMM  scope. 

Discussion:  That  portion  of  the  CMM  used  as  a  framework  for  evaluating  an  organization’s 
software  process  during  an  appraisal  is  its  CMM  scope.  It  may  include  one  or  more  maturity 
levels,  and  one  or  more  Key  Process  Areas  (KPAs)  within  each  of  these  levels.  Limiting  the 
number  of  KPAs  and/or  maturity  levels  used  in  the  appraisal  has  the  effect  of  focusing  the  in¬ 
vestigation  of  the  organization’s  software  process. 

Appraisal  method  developers  should  consider  the  following  factors  in  developing  guidance  for 
determining  an  appraisal’s  CMM  scope: 

•  The  relationship  between  CMM  scope  and  the  ability  to  provide  ratings.  An 
appraisal  team  cannot  conclude  that  a  specific  KPA  has  been  satisfied 
unless  there  has  been  an  investigation  of  all  goals  for  that  KPA.  Similarly,  an 
appraisal  team  cannot  conclude  that  a  maturity  level  has  been  achieved 
unless  there  has  been  an  investigation  of  all  KPAs  for  that  maturity  level  and 
all  lower  maturity  levels.  Without  investigating  all  KPAs  across  a  maturity 
level  or  all  goals  within  a  KPA,  a  team  can  only  conclude  that  a  maturity  level 
has  not  been  achieved  or  that  a  KPA  is  not  satisfied. 
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•  The  organization’s  software  process  maturity  ievei.  The  current  maturity  level 
of  an  organization’s  software  process  may  impact  the  range  of  findings 
required.  Suppose  an  appraisal  is  conducted  for  an  organization  with  a  level 
2  process  with  the  primary  goal  of  providing  input  to  action  plans.  Suppose 
further  that  the  appraisal  is  limited  to  investigating  only  the  KPAs  at  level  2. 

The  organization  would  then  not  have  the  data  required  to  develop 
comprehensive  action  plans  for  achieving  level  3. 

•  Constraints  on  appraisai  iength.  The  more  components  of  the  CMM  included 
in  an  appraisal’s  CMM  scope,  the  greater  the  amount  of  time  required  to 
conduct  the  appraisal  and  report  its  results. 

There  may  be  areas  outside  of  the  CMM  that  the  sponsor  is  interested  in  investigating  and 
these  areas  are  valid  scoping  considerations  for  the  appraisal.  Findings  in  these  areas  should 
be  reported,  but  in  no  way  should  they  affect  the  ratings  of  CMM  components  that  are  provided 
as  a  result  of  the  appraisal. 

3.3.1 .2.2  Organizational  Scope 

Requirement:  R5.  A  CAF  compliant  appraisal  method  shall  provide  guidance  for  choosing  an 
appraisal’s  organizational  scope. 

Discussion:  The  appraised  entity,  or  organizational  scope  of  an  appraisal,  is  the  set  of  orga¬ 
nizational  units  to  which  appraisal  outputs  apply.  The  sponsor’s  intended  use  of  the  appraisal 
results  should  drive  the  appraisal’s  organizational  scope.  For  example,  if  the  appraisal  results 
are  to  be  used  for  process  improvement,  then  the  organizational  scope  of  the  appraisal  should 
match  that  of  the  intended  improvement  effort. 

An  appraised  entity  may  be  any  portion  of  an  organization,  for  example: 

•  An  entire  company. 

•  A  selected  business  unit. 

•  A  specific  geographic  site. 

•  Organizational  units  involved  in  producing  software  for  a  particular  product 
line  or  application  domain. 

•  Organizational  units  involved  in  providing  a  particular  type  software  service 
such  as  maintenance  or  reengineering  of  legacy  systems. 

•  A  specific  project. 

•  A  team  involving  multiple  companies. 

The  CAF  does  not  require  a  method  to  impose  any  particular  organizational  constraints  in 
terms  of  structure  or  location  on  the  appraised  entity.  In  writing  this  guidance,  however,  ap¬ 
praisal  method  developers  should  consider  the  following  factors: 

Management  structure.  The  organizational  units  that  make  up  an  appraised  entity  may  have 
management  structures  with  varying  degrees  of  overlap.  For  example,  the  appraised  entity 
may  consist  of: 
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•  A  single  business  unit  with  one  business  unit  manager, 

•  Multiple  business  units  with  managers  who  report  to  a  single  executive, 

•  A  multi-company  team  with  a  project  management  structure  that  overlaps  the 
management  structures  of  each  of  the  companies  involved. 

The  extent  to  which  the  appraised  entity  shares  a  common  management  structure  has  an  im¬ 
pact  on  the  appraisal,  specifically  the  ease  of  obtaining  commitment  to  the  appraisal  process 
and  the  use  of  its  results.  If,  for  instance,  the  appraised  entity  includes  multiple  business  units, 
both  corporate  and  business  unit  managers  may  be  involved  in  the  initial  appraisal  planning 
process. 

Impact  of  shared  software  processes.  The  extent  to  which  the  appraised  entity’s  business 
goals  support  a  shared  or  common  software  process  also  impacts  the  use  of  appraisal  results. 
The  organization  may  be  involved  in  widely  varying  business  areas.  One  area  may  focus  on 
development  of  systems  to  support  major  business  reengineering  efforts  while  another  may 
focus  on  maintenance  of  existing  systems  whose  requirements  undergo  slow,  incremental 
change.  Differences  in  life  cycles,  tools,  and  techniques  employed  in  each  of  these  areas  may 
reduce  the  amount  of  commonality  in  an  organizational  level  software  process  that  is  benefi¬ 
cial.  These  differences  may  also  increase  the  complexity  of  the  tailoring  guidance  associated 
with  a  common  process. 

Organizational  and  project  level  responsibilities.  The  CMM  identifies  both  organizational  level 
and  project  level  responsibilities  for  implementation  of  KPAs.  An  appraisal’s  data  collection 
plan  must  take  into  account  the  extent  to  which  the  appraised  entity  includes  units  responsible 
for  both  project  and  organizational  level  practices.  For  example,  the  organizational  scope  of 
an  appraisal  may  be  a  single  project.  It  is  still  necessary  to  investigate  those  parts  of  the  CMM 
that  are  primarily  organizational  level  responsibilities,  such  as  establishing  policies,  defining 
the  organization’s  standard  process,  and  maintaining  the  organization’s  software  process  as¬ 
sets. 

Geographic  dispersion.  The  extent  to  which  the  appraised  entity  is  geographically  dispersed 
impacts  the  time  required  to  perform  the  appraisal  and  the  ease  with  which  the  appraisal  team 
can  maintain  its  continuity.  An  appraisal  represents  a  view  of  the  appraised  entity’s  software 
process  at  a  point  in  time.  Any  appraisal  that  spans  locations  needs  to  be  completed  in  the 
same  time  frame.  In  addition,  since  the  entire  appraisal  team  is  involved  in  making  judgments 
during  an  appraisal,  the  same  team  should  participate  in  the  appraisal  activities  at  each  loca¬ 
tion.  If  a  method  permits  different  team  members  for  appraisals  which  span  locations  then  the 
method  needs  to  address  management  of  risks  associated  with  making  consistent  team  judg¬ 
ments. 

3.3.1. 3  Establish  Commitments 

Requirement:  R6.  A  CAF  compliant  appraisal  method  shall  provide  guidance  for  obtaining  an 
organization’s  commitment  to  proceed  with  the  appraisal  process. 
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Discussion:  An  organization’s  commitment  to  perform  an  appraisal  is  essential  to  the  ability 
of  the  appraisal  team  to  produce  useful  results.  Having  senior  management  support  is  crucial 
for  the  conduct  of  the  appraisal  and  any  follow-on  activities. 

In  providing  guidance  for  the  commitment  process,  appraisal  method  developers  should  con¬ 
sider  the  following  factors: 

•  The  sponsor’s  commitment.  An  appraisal  sponsor  is  the  individual  for  whom 
an  appraisal  is  done  and  may  be  within  the  organization  or  outside  of  the 
organization.  If  an  appraisal  process  is  to  produce  results  which  meet  its 
sponsor’s  goals,  then  the  sponsor  and  the  appraisal  team  must  develop  a 
mutual  understanding  of  those  goals  and  any  associated  appraisal 
constraints, 

•  Senior  site  manager’s  commitment.  The  sponsor  of  an  appraisal  may  or  may 
not  be  the  senior  manager  in  charge  of  the  appraised  entity.  If  they  are 
different  individuals,  the  process  of  obtaining  a  commitment  to  perform  the 
appraisal  should  involve  both  the  appraisal  sponsor  and  the  appraised 
entity’s  senior  manager. 

3.3.2  Select  and  Prepare  Team 

Selecting  and  preparing  an  appraisal  team  involves  identifying  the  appraisal  team  leader,  se¬ 
lecting  each  of  the  team  members,  and  providing  the  team  with  training  and  orientation  need¬ 
ed  to  prepare  for  the  appraisal. 

3.3.2.1  Team  Qualifications 

Requirement:  R7.  A  CAP  compliant  appraisal  method  shall  document  appraisal  team  qualifi¬ 
cation  criteria  that  conform  at  a  minimum  to  the  following  technical  and  management  experi¬ 
ence  requirements: 

•  At  least  five  years  of  technical  experience  for  the  majority  of  the  team 
members. 

•  At  least  twenty  five  years  of  technical  experience  for  the  team  as  a  whole. 

•  At  least  six  years  of  management  experience  for  one  team  member. 

•  At  least  ten  years  of  management  experience  for  the  team  as  a  whole. 

Discussion:  The  composition  of  the  appraisal  team  has  a  great  impact  on  the  success  of  an 
appraisal.  Appraisal  team  selections  must  ensure  that  the  team’s  combined  knowledge,  expe¬ 
rience,  and  skills  are  appropriate  for  the  type  of  appraisal  as  well  as  the  domain  and  expected 
maturity  level  of  the  appraised  entity. 

At  the  same  time,  individuals  who  have  a  vested  interest  in  the  results  should  not  be  selected 
for  the  appraisal  team  as  their  objectivity  could  be  an  issue  and  the  focus  of  the  appraisal  may 
be  more  on  rating  considerations  than  identifying  weaknesses.  It  is,  however,  desirabie  to 
have  someone  from  the  organization  on  the  appraisal  team. 
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An  appraisal  team  needs  to  make  judgments.  The  more  experience  the  team  members  have, 
the  more  likely  they  are  to  make  consistent  judgments.  Both  technical  and  management  ex¬ 
perience  are  important  to  making  judgments.  The  breadth  of  this  experience  in  terms  of  an  ap¬ 
praised  entity’s  life  cycle  activities  is  also  important. 

For  a  team  to  exectue  an  appraisal  method  effectively,  each  team  member  must  be  trained  in 
the  method.  In  addition,  the  appraisal  team  leader  must  have  experience  in  using  the  method. 

The  team  should  also  have  knowledge  about  the  appraised  entity’s  domain  and  environment, 
and  it  is  useful  to  have  someone  from  the  organization  on  the  team.  On  the  other  hand,  infor¬ 
mation  on  the  environment  and  the  domain  could  be  supplied  to  the  team  as  part  of  an  ap¬ 
praisal  specific  orientation. 

In  addition  to  providing  appraisal  team  qualification  criteria,  appraisal  methods  should  address 
the  waiver  procedures.  Team  qualification  criteria  may  involve,  for  example,  training  that  is  un¬ 
necessary  for  people  with  particular  experience.  Waiver  procedures  allow  sponsors  and  ap¬ 
praisal  team  leaders  flexibility  in  selecting  appraisal  team  members. 

Requirement:  R8.  A  CAF  compliant  appraisal  method  shall  specify  the  qualifications  of  the 
team  leader  that  at  a  minimum  include  experience  in  using  the  appraisal  method,  managing 
teams,  facilitating  group  discussions,  and  making  presentations. 

Discussion:  The  appraisal  team  leader  needs  the  same  set  of  skills  as  the  appraisal  team 
members.  In  addition,  the  leader  must  be  an  effective  presenter,  facilitator,  and  manager  or 
must  work  with  a  coach  who  can  supplement  his/her  skills.  Sound  presentation  skills  are  need¬ 
ed  to  carry  out  orientation,  training,  and  reporting  activities.  Facilitation  skills  are  needed  to  en¬ 
sure  full  participation  in  activities  that  require  broad  organizational  participation  as  well  as 
those  that  require  team  judgment.  Management  skills  are  required  to  plan  the  appraisal  and 
to  manage  its  execution. 

3.3.2.2  Team  Size 

Requirement:  R9.  A  CAF  compliant  appraisal  method  shall  provide  guidance  for  determining 
appropriate  team  size. 

Discussion:  Team  size  affects  the  accuracy  of  appraisal  results.  Teams  may  be  too  small  to 
address  the  full  scope  of  an  appraisal  within  the  allotted  time  frame.  On  the  other  hand,  teams 
may  be  too  large  to  reach  consensus  quickly  and  efficiently  in  the  team  judgment  process. 

Observers  may  further  impact  the  efficiency  with  which  an  appraisal  team  can  conduct  ap¬ 
praisal  activities.  Appraisal  method  developers  should  provide  guidance  for  allowing  or  disal¬ 
lowing  observers  to  participate  in  the  appraisal  process.  If  observers  are  allowed,  the  guidance 
should  define  the  role  of  appraisal  observers,  clearly  identify  constraints  on  their  participation 
in  the  appraisal  process,  and  establish  limits  on  their  numbers. 
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3.3.2.3  Team  Preparation 

Requirement:  R10.  A  CAP  compliant  appraisal  method  shall  provide  guidance  for  preparing 
an  appraisal  team  to  conduct  an  appraisal,  including  applying  the  CMM. 

Discussion:  An  appraisal  team  requires  knowledge  of  the  conduct  of  the  appraisal  as  well  as 
of  the  CMM  and  the  appraised  entity.  Without  this  knowledge  the  team  cannot  effectively  eval¬ 
uate  the  entity’s  software  process  using  the  CMM  as  a  framework.  Team  members  need  suf¬ 
ficient  knowledge  to  be  able  to  explain  the  overall  structure  of  the  CMM,  the  interrelationships 
between  and  among  KPAs,  and  relate  the  CMM  to  the  appraised  entity’s  software  processes. 

In  preparing  an  appraisal  team  to  perform  a  specific  appraisal,  the  team  leader  should  consid¬ 
er  the  following  factors: 

•  Team  building.  Team  members  must  work  effectively  as  a  group  in  order  to 
complete  the  appraisal  process  in  the  allotted  time  frame.  The  appraisal 
method  should  provide  adequate  team  building  opportunity  during  the 
appraisal  planning  and  preparation  process.  Team  building  might  be 
achieved  through  joint  training  sessions  or  joint  activities  such  as  scripting 
questions  for  interview  sessions. 

•  Training.  One  way  to  close  the  gap  between  team  experience  and  the 
appraisal  method’s  minimum  team  qualifications  is  to  provide  training 
following  team  selection.  For  example,  the  appraisal  method  might  provide 
guidance  for  training  the  team  in  the  CMM  or  in  the  method’s  application. 

•  Appraisal  Specific  Orientation.  Appraisal  teams  that  are  introduced  to  the 
appraised  entity’s  way  of  doing  business  prior  to  the  appraisal  may  be  more 
effective  in  the  appraisal  process.  For  example,  an  understanding  of  the 
entity’s  terminology,  life  cycle,  application  domain,  and  environment  helps 
appraisal  team  members  phrase  questions  in  ways  that  are  meaningful  to 
appraisal  participants. 

•  Appraisal  Plan  Review.  The  appraisal  team  must  understand  and  be 
prepared  to  execute  the  appraisal  plan.  The  appraisal  method  should  provide 
mechanisms  for  achieving  this  understanding.  For  example,  the  method 
could  include  an  appraisal  plan  review  activity  or  could  involve  team 
members  in  the  planning  process  itself. 

3.3.3  Select  and  Prepare  Participants 

Appraisal  participants  are  those  people  who  provide  the  appraisal  team  with  data  concerning 
the  appraised  entity’s  software  process.  Selecting  and  preparing  an  appraisal’s  participants 
involves: 


•  Identifying  the  particular  sites,  projects,  and  organizational  support  units  from 
which  participants  will  be  drawn. 

•  Identifying  the  specific  individuals  who  will  participate  in  an  appraisal’s  data 
collection  activities. 

•  Providing  these  individuals  with  an  orientation  in  the  appraisal  process. 
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An  appraised  entity  may  span  one  or  more  geographic  locations  or  sites.  Its  software  devel¬ 
opment  and  maintenance  activities  may  be  organized  into  one  or  more  projects.  Additional  or¬ 
ganizational  units,  such  as  software  process  engineering  or  training  groups,  may  provide 
support  to  these  projects  in  accomplishing  their  activities  as  well  as  other  projects  outside  the 
scope  of  the  appraised  entity.  An  appraised  entity’s  people  may  have  affiliations  with  one  or 
more  of  its  projects  or  support  units. 

3.3.3.1  Select  Sites 

Requirement:  R1 1 .  A  CAP  compliant  appraisal  method  shall  provide  guidance  for  site  selec¬ 
tion. 

Discussion:  A  site  is  a  single  geographic  location.  The  appraised  entity  may  span  multiple 
sites.  The  sites  selected  to  participate  in  appraisal  data  collection  activities  should  be  repre¬ 
sentative  of  the  appraised  entity  so  that  appraisal  results  provide  an  accurate  picture  of  the 
appraised  entity’s  software  process. 

3.3.3.2  Select  Projects 

Requirement:  R12.  A  CAP  compliant  appraisal  method  shall  provide  guidance  for  project  se¬ 
lection. 

Discussion:  A  project  is  a  software  development  or  maintenance  effort  that  may  have  a  spe¬ 
cific  process,  budget,  schedule,  and  set  of  work  products.  The  appraised  entity  will  typically 
be  responsible  for  multiple  projects.  An  appraisal  may  focus  its  data  collection  activities  on  a 
subset  of  the  appraised  entity’s  projects.  If  these  projects  are  representative  of  the  appraised 
entity,  appraisal  results  are  more  likely  to  provide  an  accurate  picture  of  the  appraised  entity’s 
software  process.  The  projects  selected  should  be  an  appropriate  cross  section  with  respect 
to  factors  such  as  type,  size,  and  life  cycle  phase. 

Some  questions  the  appraisal  method  developers  should  consider  in  developing  project  se¬ 
lection  guidance  include: 

•  Do  the  projects  have  a  major  impact  on  achieving  the  organization’s  financial 
goals? 

•  Do  the  projects  have  strategic  importance  to  the  organization? 

•  Do  the  projects  use  technology  that  is  currently  prevalent  in  the  organization? 

•  Do  the  projects  reflect  current  practice  in  each  of  the  organization’s  life  cycle 
phases? 

•  Are  significant  differences  anticipated  in  the  maturity  of  individual  projects’ 
software  processes?  If  so,  what  impact  will  these  differences  have  on  the 
organization’s  intended  use  of  appraisal  outputs? 

3.3.3.3  Select  Individual  Participants 

Requirement:  R13.  A  CAP  compliant  appraisal  method  shall  provide  guidance  for  participant 
selection. 
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Discussion:  Participants  in  appraisal  data  collection  activities  need  to  be  representative  of 
the  entity  being  appraised.  If  participants  are  representative  of  the  appraised  entity,  then  the 
appraisal  results  are  more  likely  to  provide  an  accurate  assessment  of  the  appraised  entity’s 
software  process  capability.  The  appraisal  method  developers  should  take  into  account  the 
extent  to  which  selection  guidance  supports  a  participant  population  that  reflects: 

•  The  size  of  the  appraised  entity.  The  percentage  of  an  appraised  entity’s 
people  who  participate  in  an  appraisal  has  an  impact  on  the  appraisal  results. 

If  the  appraised  entity  is  small,  it  may  be  possible  for  all  of  its  people  to 
participate  in  the  appraisal.  If  everyone  participates  the  risk  of  selecting  a  set 
of  non-representative  set  of  appraisal  participants  is  non-existent.  Normally, 
however,  only  a  percentage  of  the  people  affiliated  with  an  appraised  entity 
can  participate  in  an  appraisal.  As  this  percentage  decreases,  the  risk  of  a 
non-representative  set  of  appraisal  participants  increases. 

•  The  breadth  of  the  appraised  entity.  At  a  minimum,  an  appraisal  should 
include  participants  from  the  sites  and  projects  selected  to  participate  in  the 
appraisal.  It  should  also  include  participants  from  organizational  level  units 
(such  as  software  process  engineering  and  training  groups)  that  provide 
support  to  these  sites  and  projects  in  carrying  out  their  activities. 

•  The  characteristics  of  the  appraised  entity’s  popuiation.  The  characteristics 
of  the  appraisal  participant  population  should  resemble  those  of  the 
appraised  entity  as  a  whole.  If  the  appraised  entity  includes  a  small 
percentage  of  very  experienced  people,  then  the  appraisal  participants 
should  also  include  only  a  small  percentage  of  very  experienced  people. 

•  A  detailed  understanding  of  the  appraised  entity’s  software  development  and 
maintenance  activities.  The  appraisal  participants,  as  a  group,  should  have 
detailed  knowledge  of  the  appraised  entity’s  software  development  and 
maintenance  activities.  They  should  also  be  aware  of  current  problems  in  the 
procedures  used  to  perform  these  activities. 

Organizational  units  other  than  projects,  such  as  software  engineering  process  groups  or  or¬ 
ganizational  level  training  groups,  may  contribute  to  a  project’s  software  development  and 
maintenance  efforts.  These  units  may  be  part  of  the  appraised  entity  or  part  of  the  larger  or¬ 
ganization  that  encompasses  the  appraised  entity.  Appraisal  method  developers  need  to  pro¬ 
vide  guidance  on  how  members  of  such  units  will  participate  in  the  appraisal  process. 

3.3.3.4  Orient  Participants 

Requirement:  R14.  A  CAP  compliant  appraisal  method  shall  provide  guidance  for  appraisal 
participant  orientation. 

Discussion:  Participants  who  understand  the  appraisal  goals,  its  planned  activities,  and  the 
anticipated  use  of  its  results  will  be  able  to  contribute  more  effectively  to  the  appraisal  process. 
An  appraisal  method  may  employ  a  variety  of  methods,  such  as  briefings  or  announcements, 
to  acquaint  participants  with  appraisal  plans.  These  preparatory  activities  may  be  targeted  at 
the  appraisal  participants  alone,  or  the  entire  organization  affected  by  appraisal  results. 
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In  providing  orientation  guidance,  appraisal  method  developers  should  consider  such  factors 
as: 

•  The  relationship  between  the  appraisal  sponsor  and  the  appraised  entity. 

•  Their  level  of  understanding  of  process  improvement  concepts. 

•  Their  level  of  CMM  knowledge. 

•  The  appraised  entity’s  prior  appraisal  experience. 

•  The  success  and/or  failure  of  its  previous  process  improvement  efforts. 

3.3.4  Develop  Appraisal  Plan 

Requirement:  R15.  A  CAP  compliant  appraisal  method  shall  provide  guidance  for  developing 
and  documenting  an  appraisal  plan  that,  at  a  minimum: 

•  Identifies  the  appraisal  goals. 

•  Identifies  the  appraisal  scope. 

•  Identifies  the  appraisal  activities. 

•  Provides  a  schedule  for  the  activities. 

•  Identifies  the  people,  resources  and  budget  required  to  perform  the  activities. 

•  Identifies  the  appraisal  outputs  and  their  anticipated  use. 

•  Identifies  anticipated  follow-on  activities. 

•  Documents  any  planned  tailoring  of  the  appraisal  method  and  associated 
trade-offs. 

•  Identifies  risks  associated  with  appraisal  execution. 

Discussion:  The  appraisal  plan  provides  a  means  to  set  expectations  concerning  the  apprais¬ 
al  process.  During  the  planning  phase,  the  appraisal  sponsor  and  the  appraisal  team  leader 
must  come  to  a  common  understanding  of  the  appraisal  process  and  its  anticipated  outcome. 
An  appraisal  plan,  when  complete  and  documented,  should  communicate  this  understanding. 

The  appraisal  method  developers  need  to  consider  addressing  the  following  topics  in  provid¬ 
ing  guidance  for  developing  and  documenting  an  appraisal  plan: 

•  Appraisal  plan  contents. 

•  Review  and  approval  of  the  appraisal  plan. 

•  Communication  of  the  plan  to  appraisal  team  and  participants. 

•  Circumstances  under  which  the  plan  should  be  updated. 

•  Mechanisms  for  review,  approval,  and  communication  of  updates. 

The  appraisal  plan  should  identify  and  document  choices  that  may  impact  the  accuracy  and 
completeness  of  appraisal  results.  The  following  are  potential  factors  that  should  be  consid¬ 
ered: 
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•  CMM  scope  of  the  appraisal. 

•  Organizational  scope  of  the  appraisal. 

•  Selected  sites,  projects,  and  participants. 

•  Selected  team  members,  team  size,  and  participation  of  observers. 

•  Planned  CMM  coverage. 

•  Appraisal  schedule,  particularly  time  allotted  for  appraisal  conduct. 

•  Availability  of  resources. 

•  Tailoring  of  the  appraisal  method. 

The  probability  of  achieving  appraisal  goals  is  related  to  the  choices  made  by  the  appraisal 
sponsor  and  the  appraisal  team  leader  in  the  planning  process.  By  documenting  the  choices, 
the  appraisal  sponsor  and  appraisal  team  leader  should  have  a  mutual  understanding  of  their 
potential  impact  on  the  success  of  the  appraisal.  Sections  3.3  and  3.5.1  include  a  discussion 
of  the  trade-offs  that  should  be  considered  in  the  appraisal  planning  process  with  respect  to 
each  of  the  factors  listed  above. 

3.3.4.1  Determine  Time  Required  to  Conduct  Appraisal 

Requirement:  R16.  A  CAP  compliant  appraisal  method  shall  provide  guidance  for  determin¬ 
ing  the  amount  of  time  required  to  conduct  an  appraisal. 

Discussion:  The  time  spent  in  conducting  the  appraisal  (collecting  data,  consolidating  it, 
making  judgments,  and  producing  appraisal  outputs)  affects  on  the  cost  of  the  appraisal,  the 
amount  of  its  disruption  to  the  appraised  entity,  and  the  quality  of  the  appraisal  outputs.  These 
effects  need  to  be  considered  in  scheduling  appraisal  activities,  particularly  those  that  involve 
the  appraised  entity’s  participants. 

The  scope  of  the  appraisal  further  influences  the  balance  that  can  be  achieved  among  ap¬ 
praisal  quality,  cost,  and  disruption.  The  greater  the  size  and  breadth  of  the  appraised  entity, 
the  more  time  required  to  achieve  confidence  in  appraisal  output.  The  more  KPAs  included  in 
the  appraisal,  the  more  time  required  as  well. 

3.3.4.2  Work  Out  Logistics 

Requirement:  R17.  A  CAP  compliant  appraisal  method  shall  provide  guidance  for  appraisal 
logistics. 

Discussion:  Appraisal  methods  should  provide  appropriate  logistics  guidance  so  logistics  do 
not  become  a  risk  to  the  appraisal’s  execution.  A  variety  of  resources  are  required  to  conduct 
appraisal  activities,  such  as  labor,  transportation,  accommodations,  facilities,  tools,  and  sup¬ 
plies.  Obtaining  these  resources  requires  commitment  from  the  appraisal  sponsor  and,  if  dif¬ 
ferent,  the  senior  manager  of  the  appraised  entity.  Scheduling  these  resources  requires  a 
certain  amount  of  coordination  among  the  appraisal  team  leader,  the  appraisal  team  mem¬ 
bers,  and  the  appraised  entity.  Appraisal  methods  should  provide  appropriate  guidance  to 
facilitate  these  interactions. 
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3.3.4.3  Select  Artifacts 

Requirement:  R18.  A  CAP  compliant  appraisal  method  shall  define  a  set  of  artifacts  to  sup¬ 
port  the  following  appraisal  activities: 

•  Recording  observations. 

•  Categorizing  observations  (with  respect  to  the  CMM). 

•  Classifying  observations  (as  strengths  or  weaknesses). 

•  Validating  observations. 

•  Recording  coverage. 

•  Making  rating  decisions. 

•  Reporting  findings  and  ratings. 

•  Managing  logistics. 

Discussion:  Forms,  templates,  and  checklists  for  appraisal  activities  help  the  team  carry  out 
the  appraisal.  They  also  serve  as  a  record  of  the  appraisal  conduct.  This  record  provides  in¬ 
formation  that  may  be  useful  in  assessing  the  effectiveness  of  the  appraisal  method.  This 
record  may  also  provide  a  level  of  detail  that  is  valuable  to  an  appraisal  team  involved  in  de¬ 
veloping  action  plans.  It  may  include  information  that  is  not,  for  confidentiality  reasons,  shared 
outside  the  team  or  included  in  the  appraisal  reports. 

3.4  Conduct  Appraisal 

As  illustrated  in  Figure  3-3,  conducting  an  appraisal  focuses  on: 

•  Collecting  and  recording  data  in  the  form  of  notes. 

•  Consolidating  data  into  a  manageable  set  of  observations,  determining  their 
validity  as  findings,  and  their  coverage  of  the  appraisal  scope. 

•  Using  those  findings  to  produce  ratings  of  the  appraised  entity’s  software 
process  with  respect  to  the  CMM. 

The  appraisal  conduct  phase  results  in  a  set  of  appraisal  outputs  which  include  both  findings 
and  ratings. 
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Figure  3-3:  Conduct  Appraisal 
3.4.1  Collect  and  Record  Data 

Requirement:  R19.  A  CAF  compliant  appraisal  method  shall  provide  guidance  for  implement¬ 
ing  data  collection  techniques  to  be  used  in  conducting  an  appraisal  and  classify  them  with 
respect  to  the  following  four  categories: 

•  Administering  instruments. 

•  Conducting  presentations. 

•  Conducting  interviews. 

•  Reviewing  documentation. 

Discussion:  The  various  sources  of  appraisal  data  fall  into  one  of  the  following  four  catego¬ 
ries: 


•  Instruments 
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•  Presentations 

•  Interviews 

•  Documents 

Data  collection  using  instruments  includes  such  activities  as  administering  questionnaires  and 
surveys  and  gathering  their  responses.  These  activities  may  or  may  not  involve  a  direct  inter¬ 
face  between  the  appraisal  team  and  the  appraisal  participants.  The  things  that  distinguish  in¬ 
struments  from  other  data  collection  mechanisms  are  the  formal,  written  nature  of  the 
information  requested  and  the  fact  that  the  people  responding  have  some  time  to  prepare  the 
response. 

Data  collection  using  presentations  can  involve  presentations  by  the  appraisal  team  or  the  ap¬ 
praisal  participants  that  include  interaction  between  the  two.  Generally,  presentations  involve 
displaying  or  demonstrating  information  about  a  predetermined  set  of  topics  to  an  audience. 
They  may  allow  appraisal  participants  to  provide  process  and  product  data  to  the  appraisal 
team  or  the  appraisal  team  to  solicit  feedback  from  the  appraisal  participants.  The  structure  of 
a  presentation  usually  restricts  the  amount  of  interchange  between  the  audience  and  the  pre¬ 
senters  to  defined  topics  and  periods  of  time. 

Data  collection  through  interviews  involves  appraisal  team  members  asking  questions  and  en¬ 
gaging  in  discussions  with  appraisal  participants  and  recording  their  responses.  Discussions 
among  appraisal  participants  facilitated  by  the  appraisal  team  also  fall  into  the  category  of  in¬ 
terviews.  Varying  numbers  of  appraisal  team  members  and  participants  may  participate  in 
such  sessions.  Participants  can  include  managers  or  technical  staff.  They  may  be  people  re¬ 
sponsible  for  performing  work  as  well  as  people  responsible  for  managing  work  performance. 
People  can  fall  into  both  categories.  For  example,  a  manager  may  oversee  the  work  per¬ 
formed  by  his/her  staff  and  still  be  responsible  for  directly  generating  certain  products  such  as 
plans  and  status  reports. 

Data  collection  using  documents  involves  reviewing  a  lasting  representation  of  information. 
Documents  may  exist  in  various  hardcopy  or  electronic  forms  such  as  bound  books,  loose  leaf 
notebooks,  slides,  posters,  word  processor  files,  data  bases,  or  electronic  mail  folders.  These 
documents  can  include  documents  that  specify  how  work  is  to  be  accomplished  (directive  doc¬ 
uments)  as  well  as  documents  that  are  records  or  products  of  the  work  performed.  Examples 
of  directive  documents  include  such  things  as  policies,  procedures,  plans,  work  instructions, 
standards,  and  checklists.  Examples  of  work  products  include  design  documents,  code,  user 
manuals,  and  analyses  of  work  records.  Examples  of  work  records  include  such  things  as  sta¬ 
tus  reports,  meeting  minutes,  and  action  item  lists.  Documents  may  fall  into  multiple  catego¬ 
ries.  For  example,  a  project  plan  that  specifies  how  the  work  to  produce  a  particular  system  is 
to  be  performed  is  a  work  product  of  a  planning  activity.  A  configuration  status  report  that  pro¬ 
vides  a  record  of  Configuration  Control  Board  decisions  may  be  a  work  product  of  the  config¬ 
uration  management  group. 
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Each  of  these  data  collection  techniques  has  inherent  strengths  and  weaknesses  in  terms  of 
the  value  of  data  that  they  provide.  Instruments  are  easy  to  replicate.  They  can  allow  partici¬ 
pants  the  flexibility  to  research  their  answers  and  to  respond  when  convenient.  They  can  pro¬ 
vide  a  means  of  comparing  the  responses  of  a  large  set  of  participants  to  the  same  set  of 
questions.  However,  instruments  do  not  provide  much  opportunity  to  eliminate  misinterpreta¬ 
tion.  Respondents  may  misinterpret  questions.  The  appraisal  team  may  misinterpret  answers. 

Presentations  can  provide  a  large  amount  of  information  to  a  large  group  of  people  in  a  short 
period  of  time.  Presentations  provide  more  opportunity  for  appraisal  team  and  appraisal  par¬ 
ticipant  interaction  than  instruments,  but  less  than  interviews.  This  interaction  decreases  the 
risk  of  misinterpretation,  although  the  presenter  may  limit  a  presentation  to  only  those  things 
he/she  chooses  to  communicate. 

Interviews  provide  the  opportunity  to  build  rapport  between  the  appraisal  team  and  the  ap¬ 
praisal  participants.  They  have  the  potential  to  facilitate  a  rapid  understanding  of  an  appraised 
entity’s  software  process.  Appraisal  teams  can  easily  adjust  the  topics  covered  during  inter¬ 
views  to  probe  areas  of  particular  strength  or  weakness.  Interviews,  however,  are  hard  to  rep¬ 
licate.  They  can  be  intimidating.  The  appraisal  team  can  have  difficulty  in  accurately  recording 
the  information  that  they  receive.  The  appraisal  team  may  ask  participants  to  describe  activ¬ 
ities  that  they  do  not  actually  perform  themselves. 

Documents  can  provide  an  objective  source  of  information.  Appraisal  teams  can  obtain  a  large 
amount  of  information  from  documents  if  they  are  well  organized  and  indexed.  Documents,  on 
the  other  hand,  can  provide  too  much  data  for  the  appraisal  team  to  absorb  in  a  short  period 
of  time.  They  also  may  be  out  of  date  or  may  have  fallen  into  disuse. 

The  data  collection  techniques  selected  and  the  manner  in  which  they  are  implemented  affect 
the  precision  of  the  information  they  provide.  For  example,  work  products  and  work  records 
would  more  likely  represent  work  actually  being  performed  as  opposed  to  a  directive  docu¬ 
ment. 

By  incorporating  a  combination  of  data  collection  techniques  in  an  appraisal,  the  appraisal 
team  can  increase  the  likelihood  of  knowing  how  work  is  actually  performed.  For  example,  an 
appraisal  team  that  did  all  of  the  following  should  have  a  better  idea  of  how  work  is  being  per¬ 
formed  than  a  team  that  did  less. 

•  Examines  a  work  product. 

•  Talks  to  the  person  or  people  who  produced  it. 

•  Examines  the  plan  that  directed  its  production. 

If  the  data  is  obtained  from  multiple  sources  and  that  data  is  consistent,  then  it  is  much  more 
likely  to  accurately  reflect  the  work  being  done  and  the  processes  implemented  than  data  ob¬ 
tained  from  a  single  source. 
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3.4.2  Consolidate  Data 

As  illustrated  in  Figure  3-4,  consolidating  data  includes: 

•  Transforming  notes  from  data  collection  sessions  into  a  set  of  observations 
and  categorizing  them  relative  to  the  CMM. 

•  Determining  that  these  observations  are  valid  findings. 

•  Determining  the  extent  to  which  the  findings  provide  adequate  coverage  of 
the  scope  of  the  appraisal  and  adjusting  data  collection  plans  accordingly. 

•  Maintaining  traceability  among  observations,  findings,  and  ultimately  ratings. 


Collect  and  Record  Data 


notes 


Figure  3-4:  Consolidate  Data 


3.4.2.1  Organize  and  Record  Data 

Requirement:  R20.  A  CAF  compliant  appraisal  method  shall  provide  guidance  for  consolidat¬ 
ing  the  data  collected  during  an  appraisal  that  addresses  at  a  minimum: 
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•  Extracting  information  from  data  gathered  during  data  collection  sessions. 

•  Recording  them  as  observations. 

•  Classifying  the  observations  (e.g.,  as  potential  strengths  or  weaknesses). 

•  Categorizing  the  observations  in  terms  of  the  CMM  or  categorizing  them  as 
non-CMM  findings. 

Discussion.  The  primary  input  to  consolidation  is  notes.  Notes  are  recordings  of  data  collec¬ 
tion  sessions.  They  are  generally  produced  by  individual  team  members.  Observations  are  in¬ 
formation  extracted  by  team  members  from  individual  notes  and  are  an  intermediate  work 
product  of  consolidation.  They  should  primarily  represent  information  concerning  an  appraised 
entity’s  software  process  that  describes  its  correspondence  with  CMM  goals  and  key  practic¬ 
es.  They  may  also  represent  information  concerning  an  appraised  entity’s  software  process 
that  is  unrelated  to  the  CMM.  The  appraisal  team  needs  to  organize  observations  in  a  manner 
that  facilitates  their  use  in  developing  appraisal  outputs,  specifically  findings  and  ratings. 

The  guidance  for  data  consolidation  should  provide  the  criteria  for  transforming  notes  into  ob¬ 
servations.  Appraisal  method  developers  should  consider  the  following  factors  in  developing 
such  guidance: 

Wording.  Valid  observations  are  findings.  Their  wording  is  an  important  element  of  communi¬ 
cating  with  the  appraisal  participants.  Clear  observations  phrased  in  terms  familiar  to  the  site 
will  be  readily  understood.  Observations  that  avoid  use  of  absolutes  such  as  “all”  or  “none”  are 
less  likely  to  be  controversial.  Observations  that  avoid  attribution  can  be  used  directly  in  ap¬ 
praisal  reports.  Wordsmithing  is  appropriate  to  ensure  the  required  confidentiality  and  to  en¬ 
sure  that  the  meaning  of  an  observation  is  correctly  conveyed.  Wordsmithing  that  changes  the 
meaning  of  recorded  data,  or  generalizes  it  without  supporting  evidence,  should  be  avoided. 

Observed  basis.  Observations  should  be  based  on  what  the  appraisal  team  hears  and  sees 
directly  and  not  on  interpretations  or  conclusions  of  what  they  have  heard  and  seen. 

Reievance.  Observations  should  be  relevant  to  the  scope  of  the  appraisal.  While  an  appraisal 
method  may  include  information  not  directly  related  to  the  areas  of  the  CMM  being  investigat¬ 
ed,  this  should  be  a  small  amount  of  information.  If  the  appraisal  does  not  focus  on  data  rele¬ 
vant  to  the  initial  scope  of  its  investigation,  reliability  of  the  appraisal  outputs  with  respect  to 
the  CMM  decreases. 

Significance.  In  order  to  reduce  notes  to  a  manageable  set  of  observations,  data  that  is  not 
significant  evidence  concerning  the  appraised  entity’s  software  process  strengths  and  weak¬ 
nesses  should  not  be  recorded  as  observations.  Generally  significant  data  will  take  one  of  the 
following  forms: 

•  Evidence  of  implementation  of  CMM  key  practices. 

•  Evidence  of  alternative  practices  which  meet  KPA  goals. 

•  Evidence  of  software  process  strengths  unrelated  to  the  CMM. 

•  Evidence  of  absence  of  CMM  key  practices. 
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•  Evidence  of  software  process  weaknesses  unrelated  to  the  CMM. 

•  Evidence  of  not  applicable  CMM  practices. 

Grouping.  Another  factor  to  consider  in  reducing  notes  to  a  manageable  set  of  observations 
is  the  manner  in  which  they  are  condensed.  Observations  may  correspond  to: 

•  A  data  item  recorded  in  notes. 

•  A  grouping  of  related  data  items. 

•  An  abstraction  or  summary  of  related  data  items. 

3.4.2.2  Validate  Data 

Requirement:  R21 .  A  CAP  compliant  appraisal  method  shall  provide  guidance  for  validating 
observations  that  includes  corroboration  of  the  observation  by  data  obtained,  at  a  minimum, 
from: 


•  Multiple  and  independent  sources. 

•  Interviews  or  presentations  that  include  people  performing  the  related  work 
or  reviews  that  include  documents  that  are  products  or  records  of  that  work. 

Discussion;  During  consolidation,  the  appraisal  team  needs  to  validate  the  observations  they 
have  recorded  to  ensure  that  they  accurately  reflect  the  practices  of  the  appraised  entity.  Find¬ 
ings  are  valid  observations  that  the  team  carries  fonward  for  subsequent  use  in  ratings  and  re¬ 
ports. 

Validation  should  be  performed  at  two  levels:  by  the  appraisal  team  and  the  appraised  entity. 
Appraisal  team  members  must  come  to  consensus  on  the  validity  of  their  observations  based 
on  the  data  that  they  collect.  In  addition,  the  appraisal  team  should  solicit  feedback  on  these 
observations  from  the  appraised  entity.  They  can  do  this  directly  through  such  mechanisms 
as  draft  findings  presentations  or  indirectly  through  such  means  as  consolidation  interviews 
designed  to  test  their  assumptions  or  both.  In  some  applications,  validation  by  the  appraised 
entity  may  not  be  possible. 

Validation  guidance  needs  to  address  the  manner  in  which  the  appraisal  team  corroborates 
observations.  The  rationale  for  the  minimum  guidance  outlined  above  is  as  follows: 

•  Observations  should  come  from  multiple  and  independent  sources  to  ensure 
that  they  do  not  reflect  just  one  person’s  perspective  concerning  the 
appraised  entity’s  processes.  Participants  in  presentations  and  interviews 
may  be  biased  by  the  comments  others  make;  therefore,  observations 
should  also  be  supported  by  data  from  multiple  data  collection  sessions. 

•  Observations  should  be  confirmed  by  people  performing  related  work  or  by 
documents  that  are  the  output  or  record  of  that  work  because: 

•  Observations  concerning  a  particular  activity  are  more  likely  to  be 
accurate  if  they  are  based  on  data  provided  by  the  people  performing  the 
activity  rather  than  people  managing  or  observing  the  activity  from  a 
distance. 
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•  Observations  concerning  a  particular  activity  are  more  likely  to  be 
accurate  if  they  are  based  on  a  review  of  the  activity’s  work  product  and 
work  records  rather  than  directive  documents  that  indicate  how  the  work 
should  be  performed. 

In  providing  validation  guidance,  appraisal  method  developers  should  also  consider  the  need 
to  resolve  conflicts  that  might  arise  between  observations  recorded  by  different  appraisal  team 
members.  Conflicts  might  include: 

•  Evidence  of  weaknesses  and  non-applicability  related  to  the  same  CMM 
practice. 

•  Evidence  of  weaknesses  and  alternative  implementations  associated  with 
the  same  CMM  practice. 

•  Evidence  of  weaknesses  and  strengths  associated  with  the  same  CMM 
practice. 

The  conflicts  might  arise  for  a  number  of  reasons  such  as: 

•  Team  members  may  review  different  documents. 

•  Team  members  may  make  observations  that  the  team  later  discovers  pertain 
only  to  a  subset  of  the  appraised  entity. 

•  Team  members  may  record  weaknesses  prior  to  discovering  that  the  related 
practices  are  not  applicable  in  the  organization’s  environment. 

•  Team  members  may  record  weaknesses  prior  to  discovering  that  an 
acceptable  alternative  to  a  CMM  practice  has  been  implemented. 

Requirement:  R22.  A  CAP  compliant  appraisal  method  shall  provide  guidance  for  validating 
observations  that  requires  at  a  minimum  that  a  portion  of  the  observations  related  to  each  of 
the  KPA  goals  within  the  scope  of  the  appraisal  are  supported  by  a  review  of  related  documen¬ 
tation. 

Discussion:  Please  note  that  the  key  practices  to  goal  mapping  is  shown  in  Appendix  B.  The 
key  practices  elaborate  on  the  goals  and  provide  guidance  for  interpreting  them,  but  they  are 
not  mandated,  as  there  may  be  alternative  implementations  that  meet  the  goals’  intent. 

In  order  to  ensure  that  the  appraisal  teams’  perception  of  the  appraised  entity’s  practices  is 
accurate,  a  significant  portion  of  the  observations  should  be  confirmed  by  a  review  of  docu¬ 
mentation.  It  is  a  method  specific  implementation  to  specify  that  amount. 

Review  of  documentation  is  primarily  directed  at  identifying  strengths  as  a  lack  of  documenta¬ 
tion  is  often  a  confirmation  of  a  weakness. 

The  purpose  of  reviewing  documentation  is  not  limited  to  confirming  existing  observations. 
Observations  can  be  generated  based  on  document  review,  which  would  still  need  to  be  inde¬ 
pendently  validated. 
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3.4.2.3  Determine  Coverage 

Requirement:  R23.  A  CAP  compliant  appraisal  method  shall  provide  guidance  for  evaluating 
and  documenting  the  sufficiency  of  findings  relative  to  the  scope  of  the  appraisal,  including 
coverage  criteria  that  adhere  to  the  following  minimum  set  of  rules  for  determining  whether 
sufficient  data  exists  to  support  rating: 

•  A  goal  is  covered  if  sufficient  findings  exist  to  judge  the  extent  of  its 
implementation  and  institutionalization  relative  to  the  CMM,  the  appraised 
entity,  and  the  appraised  entity’s  life  cycle(s)  (including  the  existence  of 
acceptable  alternatives). 

•  A  Key  Process  Area  is  covered  if  all  of  its  goals  are  covered. 

•  A  maturity  level  is  covered  if  all  of  its  Key  Process  Areas  and  all  those  of 
lower  level  KPAs  are  covered. 

Discussion:  Rigorous  investigation  of  every  key  practice  may  not  be  feasible  during  the 
course  of  an  appraisal.  A  CAP  compliant  appraisal  method  needs  to  establish  sampling  criteria 
for  both  the  activities  performed  as  well  as  the  institutionalization  common  features.  This  may 
involve  investigating  every  key  practice,  investigating  a  random  percentage  of  key  practices, 
investigating  a  specific  subset  of  key  practices,  or  some  other  criteria. 

The  reliability  of  an  appraisal’s  outputs  is  directly  related  to  the  extent  to  which  an  appraisal’s 
findings  provide  an  adequate  sampling  of  key  practices  and  software  process  data  related  to 
the  scope  of  the  appraisal.  This  includes: 

•  Coverage  of  the  components  of  the  CMM  selected  for  investigation. 

•  Coverage  of  the  appraised  entity. 

•  Coverage  of  the  appraised  entity’s  life  cycle  activities. 

Appraisal  method  developers  should  consider  the  following  factors  in  describing  rules  for  gath¬ 
ering  and  analyzing  coverage  information: 

Adequate  CMM  coverage.  Satisfaction  of  a  KPA  depends  on  satisfaction  of  the  goals.  This 
satisfaction  can  involve  implementation  of  the  key  practices  that  map  to  that  goal  as  described 
in  Appendix  B  or  implementation  of  an  alternative  set  of  practices  that  achieve  the  goal.  A  CAP 
compliant  method  requires  coverage  of  all  of  the  goals  that  fall  within  the  scope  of  the  apprais¬ 
al.  A  CAP  compliant  appraisal  method  requires  a  sampling  of  the  appraised  entity’s  key  prac¬ 
tices  that  map  to  each  goal. 

Adequate  organizational  coverage.  During  the  appraisal  planning  process,  a  representative 
sample  of  sites,  projects,  and  people  are  selected  to  participate  in  the  appraisal.  During  ap¬ 
praisal  conduct,  it  is  incumbent  upon  the  appraisal  team  to  ensure  that  data  actually  collected 
covers  each  of  these  organizational  entities.  Documents  related  to  each  of  the  selected 
projects  should  be  sampled.  Data  should  be  solicited  from  each  of  the  appraisal  participants. 
The  appraisal  team  must  be  capable  of  facilitating  full  participation  in  group  interview  and  pre¬ 
sentation  sessions. 
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Adequate  life  cycle  coverage.  Some  software  process  activities  are  limited  to  particular  phas¬ 
es  of  an  appraised  entity’s  life  cycle,  while  others  span  multiple  life  cycle  phases.  The  apprais¬ 
al  team  must  ensure  that  the  data  collected  covers  each  of  the  life  cycle  activities  in  which  the 
appraised  entity  is  involved.  The  team  may  need  to  focus  the  topics  explored  during  inter¬ 
views,  presentations,  and  document  review  on  specific  life  cycle  phases. 

Extension  to  coverage  rules.  Appraisal  method  developers  may  choose  to  impose  a  more 
stringent  set  of  rules  for  determining  coverage.  For  example,  an  appraisal  method  might  re¬ 
quire  that  the  findings  related  to  each  individual  key  practice  cover  all  organizational  units 
comprising  the  appraised  entity  or  all  applicable  life  cycle  phases.  Such  extensions  in  cover¬ 
age  rules  should  be  clearly  related  to  the  appraisal  goals  since  they  will  lengthen  the  appraisal 
schedule  and  increase  both  the  cost  and  disruption  associated  with  the  appraisal. 

Recording  Coverage  Data.  Coverage  data  quantifies  the  degree  to  which  the  appraisal  team’s 
findings  map  to  or  cover  elements  of  the  CMM,  the  appraised  entity,  and  the  associated  life 
cycle  activities  that  fall  within  the  scope  of  an  appraisal.  Generally,  the  appraisal  team  will  use 
coverage  data  to  track  progress  towards  their  goal  of  developing  a  sufficient  set  of  findings. 
The  appraisal  team  must  protect  the  confidentiality  of  this  data.  Appraisal  reports  may  include 
only  summaries  of  coverage  data  that  do  not  indicate  attribution  of  information  to  particular 
sources. 

Requirement:  R24.  A  CAF  compliant  appraisal  method  shall  define  mechanisms  for  adjusting 
data  collection  plans  to  obtain  sufficient  coverage. 

Discussion:  CAF  compliant  appraisal  methods  involve  multiple  data  collection  sessions.  Ad¬ 
justments  to  the  schedule  for  such  sessions  may  be  necessary  in  order  to  obtain  sufficient  cov¬ 
erage  of  the  appraisal’s  scope.  These  adjustments  can  occur  for  any  number  of  reasons. 
Participants  in  a  given  interview  may  not  have  the  responsibilities  or  knowledge  of  activities 
assumed  by  the  appraisal  team.  The  appraisal  team  may  not  complete  interviews  in  the  allot¬ 
ted  time  frame.  The  questions  asked  may  not  provide  the  depth  of  information  required  to  un¬ 
derstand  the  extent  to  which  a  particular  practice  is  implemented. 

3.4.2.4  Maintain  Traceability 

Requirement:  R25.  A  CAF  compliant  appraisal  method  shall  provide  guidance  for  recording 
traceability  between  the  data  collected  during  the  appraisal  and  the  appraisal  outputs. 

Discussion:  Traceability  allows  the  appraisal  team  to  determine  the  source  of  the  data  on 
which  its  intermediate  and  final  work  products  are  based.  Tracking  data  may  be  critical  to  re¬ 
solving  questions  that  arise  during  team  discussions  regarding  consolidation  or  rating.  It  may 
also  be  necessary  to  evaluate  feedback  provided  during  presentations  or  to  resolve  questions 
about  the  final  reports.  It  may  be  useful  in  subsequent  action  planning.  Without  such  data  it 
may  be  difficult  for  the  appraisal  team  to  explain  and  discuss  findings  and  recommendations 
with  the  appraised  entity. 


CMU/SEI-95-TR-001 


35 


Tracking  data,  however,  allows  individuals  to  attribute  appraisal  findings  and  ratings  to  infor¬ 
mation  provided  by  specific  people,  projects,  and  sites.  The  appraisal  team  must  protect  the 
confidentiality  of  this  data.  While  it  provides  a  resource  to  the  appraisal  team  that  allows  them 
to  conduct  their  appraisal  efficiently,  it  generally  cannot  be  shared  outside  the  team.  Excep¬ 
tions  to  this  general  rule  may  include  appraisals  that  by  agreement  of  the  appraisal  team  lead¬ 
er,  sponsor,  and  senior  site  manager(s)  produce  project  or  site  specific  findings.  Such 
appraisals  must  still  avoid  attribution  of  findings  to  data  provided  by  specific  people.  Appraisal 
teams  may  elect  to  destroy  their  notes  after  the  appraisal  to  further  protect  the  confidentiality 
of  the  appraisal  participants  who  provided  the  data  and  retain  only  the  data  that  is  non-attrib- 
utable. 

Tracking  data  should  show  the  correspondence  among  appraisal  data  sources,  observations, 
findings,  and  ratings.  This  data  can  be  recorded  using  pointers,  references,  traceability  matri¬ 
ces,  or  other  simple  record  keeping  methods.  The  appraisal  team  should  be  capable  of  record¬ 
ing  this  data  with  minimal  effort  and  time.  Table  3-1  shows  an  example  of  a  simple  traceability 
matrix. 


I  Rating 

Finding 

I  Observation 

I  Session 

I  Source  I 

Rating  1 

Finding  1 

Observation  y 

Document  review 

Document  1,  pg  12 

Interview  1 

Document  2,  pg  15 

Interview  6 

Software  engineer  2 

Finding  2 

Observation  x 

Interview  2 

Project  manager  2 

Interview  8 

Software  designer  2 

Table  3-1 :  Sample  Traceability  Matrix 

3.4.3  Make  Rating  Judgments 
3.4.3.1  Rated  Components 

Requirement:  R26.  A  CAP  compliant  appraisal  method  shall  require  rating  of  the  following 
categories  of  CMM  components  provided  they  fall  within  the  scope  of  the  appraisal: 

•  KPAs 

•  Goals 

Discussion:  Figure  3-5  illustrates  the  rated  components  within  the  structure  of  the  CMM. 
There  are  three  components  of  the  CMM  reference  model  that  can  be  rated:  goals,  KPAs,  and 
maturity  level.  Maturity  level  ratings  are  optional,  since  the  rating  itself  provides  minimal  visi¬ 
bility  into  the  state  of  an  appraised  entity’s  software  process.  However,  all  goals  and  KPAs 
within  the  appraisal  scope  should  be  rated.  Appraisal  method  developers  may  choose  to  ex¬ 
tend  the  rating  scheme  and  rate  common  features  and/or  key  practices. 
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=  Rated  component 


Figure  3-5:  Rated  Components  Within  the  Structure  of  the  CMM 
3.4.3.2  Rating  Scale 

Requirement:  R27.  A  CAF  compliant  appraisal  method,  if  it  calculates  a  maturity  level  rating, 
shall  require  that  the  maturity  level  rating  is  consistent  with  the  five  level  scale  provided  in  the 
CMM. 

Discussion:  The  maturity  level  rating  scale  consists  of  five  integral  levels  (1  -  5)  as  described 
in  the  CMM. 

Requirement:  R28.  A  CAF  compliant  appraisal  method  shall  define  a  rating  process  using  the 
following  rating  values: 

•  A  KPA  or  goal  is  satisfied  if  this  aspect  of  the  CMM  is  implemented  and 
institutionalized  either  as  defined  in  the  CMM,  or  with  an  adequate 
alternative. 

•  A  KPA  or  goal  is  unsatisfied  if  there  are  significant  weaknesses  in  the 
appraised  entity’s  implementation  or  institutionalization  of  this  aspect  of  the 
CMM,  as  defined,  and  no  adequate  alternative  is  in  place. 

•  A  KPA  or  goal  is  not  applicable  if  the  KPA  is  not  applicable  in  the 
organization’s  environment. 

•  A  KPA  or  goal  is  not  rated  if  the  associated  appraisal  findings  do  not  meet 
coverage  criteria  or  if  this  aspect  of  the  CMM  falls  outside  the  scope  of  the 
appraisal. 
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Discussion:  Four  rating  values  are  provided  for  goals  and  KPAs:  satisfied,  unsatisfied,  not 
applicable,  or  not  rated.  If  a  KPA  is  determined  to  be  not  applicable  in  the  organization’s  envi¬ 
ronment,  then  all  of  the  goals  for  that  KPA  are  deemed  not  applicable.  Conversely,  if  a  KPA  is 
determined  to  be  applicable  in  the  organization’s  environment,  then  all  of  the  goals  for  that 
KPA  are  applicable. 

Appraisal  method  developers  may  add  additional  ratings  such  as  partially  satisfied,  to  this  rat¬ 
ing  scale.  A  partially  satisfied  rating  maps  to  not  satisfied  on  a  satisfied/not  satisfied  rating 
scale. 

3.4.3.3  When  Ratings  Can  Be  Performed 

Requirement:  R29.  A  CAF  compliant  appraisal  method  shall  define  a  rating  process  which 
specifies  that: 

•  An  appraisal  team  can  rate  a  goal  when  valid  observations  related  to  the  goal 
meet  the  method’s  defined  coverage  criteria. 

•  An  appraisal  team  can  rate  a  KPA  when  it  has  rated  each  of  the  associated 
goals. 

•  An  appraisal  team  can  determine  a  maturity  level  rating  once  it  has  rated  all 
of  the  KPAs  within  that  level  and  each  level  below. 

Discussion:  The  appraisal  team  develops  ratings  by  making  judgments  based  on  the  data 
that  they  collect  during  the  appraisal  process.  The  appraisal  team’s  findings  relative  to  a  com¬ 
ponent  of  the  KPA  must  meet  that  component’s  coverage  criteria  (see  Section  3.4.2.3)  in  order 
for  an  appraisal  team  to  provide  the  component  rating.  It  should  be  rare  that  an  appraisal  team 
does  not  get  sufficient  data  to  be  able  to  derive  a  rating.  The  team  must  make  every  effort  to 
acquire  sufficient  data.  However,  it  is  better  for  an  appraisal  team  to  conclude  without  a  rating 
than  to  derive  one  with  insufficient  data. 

As  illustrated  in  Figure  3-6,  ratings  are  determined  in  a  hierarchical  fashion. 


Figure  3-6:  Rating  Sequence 

The  rating  of  any  given  CMM  component  is  dependent  on  ratings  of  the  more  detailed  CMM 
components,  if  any.  In  particular: 

•  Maturity  level  ratings  depend  exclusively  on  KPA  ratings.  For  example,  rating 
of  maturity  level  3  requires  that  all  KPAs  within  levels  2  and  3  be  satisfied  or 
not  applicable. 

•  KPA  ratings  depend  on  the  ratings  of  the  goals.  An  appraisal  team  cannot 
rate  a  KPA  satisfied  if  any  goals  are  unsatisfied. 
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3.4.3.4  Rating  Algorithm 

Requirement:  R30.  A  CAP  compliant  appraisal  method  shall  define  a  rating  process  that  re¬ 
quires  full  participation  of  all  appraisal  team  members  in  reaching  consensus  in  all  rating  de¬ 
cisions. 

Discussion:  The  appraisal  team  must  come  to  consensus  on  the  ratings  that  it  provides  to  an 
appraised  entity.  This  consensus  is  one  step  in  assuring  that  the  entire  team  supports  the  ap¬ 
praisal  report.  Without  consensus,  the  appraisal  team  cannot  expect  the  appraised  entity  to 
have  a  high  level  of  confidence  in  the  contents  of  the  report. 

Requirement:  R31 .  A  CAP  compliant  appraisal  method  shall  require  that  ratings  be  based  on 
the  CMM  as  defined  in  Capability  Maturity  Model  for  Software,  Version  1.1  [Paulk  93a]  and 
Key  Practices  of  the  Capability  Maturity  Model,  Version  1. 1  [Paulk  93b]. 

Discussion:  All  ratings  must  be  based  on  the  CMM  and  only  on  the  CMM.  A  CMM-based  ap¬ 
praisal,  by  definition,  is  using  the  CMM  as  a  framework  for  evaluating  an  appraised  entity’s 
software  process.  The  appraisal  team  must,  therefore,  maintain  fidelity  to  the  model  in  its  rat¬ 
ing  process.  An  appraisal  method  cannot  add  new  KPAs  to  the  model  or  delete  existing  KPAs. 

Requirement:  R32.  A  CAP  compliant  appraisal  method  shall  define  a  rating  process  that  re¬ 
quires  appraisal  teams  to  base  ratings  on  their  findings  (observations  that  they  have  validat¬ 
ed). 

Discussion:  Ratings  must  be  based  on  the  data  the  appraisal  team  collects  during  the  ap¬ 
praisal  process.  By  basing  ratings  on  findings  that  have  been  validated  by  the  team,  and  di¬ 
rectly  or  Indirectly  by  the  appraised  entity,  the  appraisal  team  can  achieve  a  high  level  of 
confidence  in  their  accuracy. 

3.4.3.4.1  Judge  Satisfaction  of  Goals 

Requirement:  R33.  A  CAP  compliant  appraisal  method  shall  define  a  rating  process  which 
specifies  that  each  goal  is  rated  in  accordance  with  the  following  rules: 

•  Rate  the  goal  “satisfied”  if  the  associated  findings  indicate  that  this  goal  is 
implemented  and  institutionalized  either  as  defined  in  the  CMM  with  no 
significant  weaknesses  or  that  an  adequate  alternative  exists. 

•  Rate  the  goal  “unsatisfied”  if  the  associated  findings  indicate  that  there  are 
significant  weaknesses  in  the  appraised  entity’s  implementation  and 
institutionalization  of  this  goal  as  defined  in  the  CMM  and  no  adequate 
alternative  is  in  place. 

•  Rate  the  goal  “not  applicable”  if  the  KPA  is  not  applicable  in  the 
organization’s  environment. 

•  Rate  the  goal  “not  rated”  if  the  associated  findings  do  not  meet  the  method’s 
defined  criteria  for  coverage  or  if  the  goal  falls  outside  of  the  scope  of  the 
appraisal. 
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Discussion:  The  CMM  is  a  framework  with  defined  but  not  prescribed  practices.  CMM  spec¬ 
ified  key  practices  may  not  be  applicable  in  the  context  of  an  appraised  entity’s  business.  It  is 
also  possible  that  the  appraised  entity  has  implemented  an  alternate  set  of  practices  that 
achieve  the  intent  and  purpose  of  the  associated  KPA  goals. 

Furthermore,  it  is  unlikely  that  an  appraised  entity  will  implement  all  CMM  key  practices  and 
subpractices  on  all  projects  all  of  the  time.  Weaknesses  in  the  implementation  of  CMM  prac¬ 
tices  may  or  may  not  be  significant  when  looking  at  the  appraised  entity  as  a  whole.  However, 
for  a  rating  of  “satisfied,”  if  there  are  identified  weaknesses  they  would  have  to  be  judged  in¬ 
significant  in  achieving  the  KPA  goal. 

3.4.3.4.2  Judge  Satisfaction  of  KPAs 

Requirement:  R34.  A  CAF  compliant  appraisal  method  shall  define  a  rating  process  which 
specifies  that  each  KPA  is  rated  in  accordance  with  the  following  rules: 

•  Rate  the  KPA  “satisfied”  if  all  of  the  goals  are  rated  “satisfied.” 

•  Rate  the  KPA  “unsatisfied”  if  any  goal  is  rated  as  “unsatisfied.” 

•  Rate  the  KPA  “not  applicable”  if  the  KPA  is  not  applicable  in  the 
organization’s  environment. 

•  Rate  the  KPA  “not  rated”  if  any  of  the  goals  are  rated  “not  rated”  or  if  the  KPA 
falls  outside  of  the  scope  of  the  appraisal. 

Discussion:  Satisfaction  of  KPAs  is  based  solely  on  the  satisfaction  of  the  goals.  If  any  one 
of  the  goals  is  not  satisfied,  the  KPA  is  automatically  not  satisfied. 

3.4.3.4.3  Determine  Maturity  Level 

Requirement:  R35.  A  CAF  compliant  appraisal  method  shall  define  a  rating  process  which 
specifies  that  maturity  level,  if  determined,  is  determined  in  accordance  with  the  following 
rules: 


•  A  maturity  level  is  satisfied  if  all  KPAs  within  that  level  and  each  lower  level 
are  satisfied  or  not  applicable. 

•  The  maturity  level  rating  is  that  of  the  highest  maturity  level  satisfied. 

Discussion:  The  appraisal  team  bases  maturity  level  ratings  solely  on  the  KPA  ratings.  No 
additional  team  judgments  are  required. 

3.5  Report  Results 

As  illustrated  in  Figure  3-7,  the  Reporting  Phase  of  an  appraisal  involves: 

•  Reporting  appraisal  results  to  sponsors,  the  appraisal  method  owner,  the 
SEI,  and,  optionally,  the  appraised  entity. 

•  Preserving  appraisal  records. 

Protecting  confidentiality  of  appraisal  data  is  an  important  aspect  of  both  of  these  activities. 
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Plan  and  Prepare  for  Appraisal  Conduct  Appraisal 


Figure  3-7:  Report  Results 


3.5.1  Report  Appraisal  Results 

Requirement:  R36.  A  CAF  compliant  appraisal  method  shall  identify  the  reports  that  the  ap¬ 
praisal  team  provides  as  the  result  of  the  appraisal  process. 

Discussion:  Appraisal  methods  should  specify  the  types  of  reports  that  the  appraisal  team 
should  provide  based  on  the  appraisal  goals.  At  a  minimum  these  should  include  reports  to 
sponsors  and  the  SEI.  Every  CAF  compliant  appraisal  method  should  report  the  results  to  the 
appraised  entity  for  use  in  process  improvement;  however,  there  may  be  instances  when  this 
is  not  feasible.  If  the  appraisal  method  was  developed  outside  of  the  SEI,  a  report  of  the  ap¬ 
praisal  conduct  should  be  provided  to  the  method  owner. 

Requirement:  R37.  A  CAF  compliant  appraisal  method  shall  require  that  the  appraisal  team 
report  the  following  data,  at  a  minimum,  to  the  sponsor: 

•  Appraisal  scope. 

•  Appraisal  selections  (site,  projects,  participants,  team  members). 

•  Findings. 
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•  Ratings. 

•  Risks  associated  with  the  accuracy  and  completeness  of  appraisal  outputs. 

Discussion:  Reports  to  sponsors  include  a  restatement  of  the  appraisal  scope  in  order  to 
identify  the  domain  to  which  the  findings  and  ratings  are  applicable.  They  include  a  description 
of  the  selections  made  during  appraisal  planning  in  order  to  provide  an  understanding  of  the 
breadth  and  depth  of  the  data  provided  by  the  appraisal  and  the  qualifications  of  the  appraisal 
team.  Findings  and  ratings  are  included,  of  course,  since  they  are  the  primary  output  of  the 
appraisal  process  and  the  data  that  forms  the  basis  for  follow-on  activities.  The  appraisal  re¬ 
port  should  identify  the  activities  that  will  follow  an  appraisal  and  the  manner  in  which  they  will 
use  appraisal  outputs.  Appraisal  teams  need  to  understand  the  relationship  between  the  ap¬ 
praisal  and  anticipated  follow-on  activities  so  that  they  provide  the  data  required  to  implement 
the  follow-on  activities.  They  also  need  to  ensure  that  actions  taken  during  the  appraisal  pro¬ 
cess  facilitate  rather  than  impede  the  effectiveness  of  follow-on  activities. 

Appraisal  outputs  may  be  a  significant  factor  in  the  decisions  made  by  appraisal  sponsors 
and/or  appraised  entities.  They  may  result  in  selection  of  one  software  supplier  over  another. 
They  may  guide  the  areas  in  which  an  organization  focuses  its  software  process  improvement 
efforts.  Inaccurate  findings  may  lead  to  inappropriate  business  decisions.  Incomplete  findings 
may  result  in  lost  opportunities  to  address  significant  areas  of  weakness  or  build  on  significant 
areas  of  strength. 

While  there  are  many  different  risk  factors  that  impact  the  accuracy  and  completeness  of  ap¬ 
praisal  outputs,  SEI  experience  with  existing  appraisal  methods  indicates  that  the  factors  iden¬ 
tified  below  are  key. 

•  Appraisal  scope.  Factors  associated  with  an  appraisal’s  scope  are  identified 
in  Section  3.3.1 .2  and  include  those  associated  with  selecting  an  appraisal’s 
CMM  scope  as  well  as  its  organizational  scope. 

•  Appraisal  team.  Factors  associated  with  team  selection  are  described  in 
Section  3.3.2  and  include  those  associated  with  team  qualifications,  size,  the 
use  of  observers,  and  the  objectivity  of  the  team. 

•  Appraisal  sample.  Factors  associated  with  selecting  sites,  projects  and 
participants  are  addressed  in  Section  3.3.3  and  include  those  associated 
with  obtaining  a  representative  sample  of  sites,  projects,  organizational 
support  units,  and  people  to  participate  in  the  appraisal  process  as  well  as 
the  appraisal  participants  being  entirely  forthcoming. 

•  Adequate  coverage.  Factors  associated  with  obtaining  adequate  coverage  of 
the  appraisal  scope  are  discussed  in  Section  3.4.2.3  and  include  those 
associated  with  coverage  of  CMM,  the  appraised  entity,  and  the  entity’s  life 
cycle(s).  In  particular,  there  may  be  risks  with  sampling  CMM  components  as 
opposed  to  obtaining  full  coverage,  and  the  actual  coverage  of  CMM 
components  should  be  reported. 
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•  Validation  of  data.  Factors  associated  with  validating  findings  are  described 
in  Section  3.4.2.2  and  include  those  associated  with  corroboration  of 
appraisal  team  observations  as  well  as  risks  associated  with  eliciting 
feedback  regarding  these  observations  from  the  appraised  entity. 

•  Appraisal  constraints.  Factors  associated  with  appraisal 
constraints — schedule,  cost,  and  disruption — are  discussed  in  Section 
3.3.4.1. 

•  Adherence  to  appraisal  method  guidance.  Appraisal  plans  must  be 
developed  to  conform  to  an  appraisal  method’s  guidance.  Teams  must 
successfully  execute  these  plans.  Deviations  from  appraisal  method 
guidance  and/or  appraisal  plans  can  have  a  significant  impact  on  appraisal 
outputs. 

Risks  associated  with  the  accuracy  and  completeness  of  appraisal  outputs  are  primarily  influ¬ 
enced  by  three  things: 

•  The  choices  made  by  method  developers  in  implementing  a  CAF  compliant 
appraisal  method. 

•  The  choices  made  by  the  appraisal  sponsor  and  appraisal  team  leader  in 
planning  an  appraisal. 

•  The  extent  to  which  the  appraisal  team  adheres  to  the  appraisal  plan. 

Reports  provided  to  the  sponsor  must  observe  the  level  of  confidentiality  agreed  to  for  this  ap¬ 
praisal.  Any  additional  distribution  of  these  reports  is  made  solely  at  the  discretion  of  the  spon¬ 
sor,  including  distribution  to  members  of  the  appraised  entity. 

Requirement:  R38.  A  CAF  compliant  appraisal  method  shall  report  appraisal  results  to  the 
SEI. 

Reports  to  the  SEI  provide  information  required  to  determine  the  state  of  software  process  in¬ 
dustry-wide  and  to  improve  the  CMM.  This  information  should  include  appraisal  plans  and  re¬ 
ports,  details  concerning  interpretation  of  the  CMM,  applicability  of  its  practices,  and  the 
existence  of  acceptable  alternative  practices. 

The  report  should  additionally  provide  information  required  to  improve  appraisal  methods  such 
as  descriptions  of  appraisal  method  problems,  suggestions  for  improvements,  and  information 
regarding  tools  that  automate  appraisal  processes.  If  the  method  was  developed  outside  of 
the  SEI,  this  additional  information  should  also  be  provided  to  the  method  owner. 

3.5.2  Protect  Confidentiality 

Requirement:  R39.  A  CAF  compliant  appraisal  method  shall  provide  guidance  for  protecting 
the  confidentiality  of  appraisal  information. 

Discussion:  In  providing  guidance  for  protecting  the  confidentiality  of  appraisal  information, 
method  developers  should  consider  the  following  factors: 
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•  Attribution  in  appraisai  reports.  In  planning  and  preparing  for  an  appraisal, 
the  appraisal  sponsor  and  appraisal  team  leader  agree  on  the  degree  of 
attribution  allowed  in  appraisal  results.  Attribution  to  individuals  is  not  allowed 
since  it  would  significantly  impact  the  risks  associated  with  obtaining 
complete  and  accurate  data.  Attribution  to  sites  and  projects  may  be  allowed 
with  agreement  of  site  and  project  managers  depending  on  appraisal  goals. 

•  Sharing  of  appraisal  reports.  The  appraisal  team  should  not  distribute  reports 
to  any  party  other  than  the  appraisal  sponsor  and  the  SEI  without  the 
sponsor’s  permission.  It  may,  for  example,  not  be  in  the  best  interests  of  an 
organization  to  share  data  concerning  its  software  process  with  competitors. 

Even  distribution  of  reports  to  members  of  the  appraised  entity  may  have 
significant  legal  implications  if  the  reports  are  related  to  appraisals  performed 
to  select  software  suppliers. 

•  Informal  communication  of  appraisal  data.  Multiple  people  may  participate  in 
data  collection  activities.  Information  exchanged  during  those  activities  must 
be  protected  both  by  the  appraisal  team  and  by  the  appraisal  participants. 

Appraisal  records  may  be  maintained  for  use  in  subsequent  action  planning. 

The  appraisal  team  must  take  precautions  to  ensure  use  of  these  records 
does  not  violate  the  confidentiality  agreements  established  during  appraisal 
planning. 

3.5.3  Preserve  Records 

Requirement:  R40.  A  CAP  compliant  appraisal  method  shall  provide  guidance  for  retention 
of  appraisal  records. 

Discussion:  Appraisal  method  developers  should  consider  the  following  factors  in  writing  this 
guidance: 


•  Purpose.  Appraisal  records  should  only  be  preserved  if  they  are  likely  to  be 
used  in  post  appraisal  activities.  For  example,  the  appraisal  plan  and  reports 
may  be  useful  in  conducting  a  subsequent  appraisal  even  after  a  period  of 
one  or  two  years.  More  detailed  records,  such  as  notes,  observations, 
findings,  ratings,  and  associated  tracking  and  coverage  data  may  be  required 
to  permit  a  post  appraisal  audit  for  those  appraisals  that  are  components  of 
acquisition  activities.  They  may  also  be  useful  in  the  generation  of  detailed 
action  plans. 

•  Safeguards.  The  confidentiality  agreements  established  during  the  appraisal 
planning  process  remain  in  effect  indefinitely.  Appraisal  records  must  be 
safeguarded  to  ensure  that  confidentiality  is  maintained. 

•  Logistics.  The  appraisal  method  should  address  the  logistics  associated  with 
preserving  appraisal  data,  controlling  access  to  it,  and  disposing  of  it. 
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Appendix  A  Requirements 


Requirement:  R1 .  A  CAP  compliant  appraisal  method  shall  be  documented,  including  at  a 
minimum: 


•  Identifying  the  version  of  the  CMM  and  the  CAP  on  which  it  depends. 

•  Documenting  the  manner  in  which  it  has  implemented  appraisal  method 
activities,  artifacts  and  guidance  required  by  the  CAP. 

Requirement:  R2.  A  CAP  compliant  method  shall  provide  guidance  for  three  phases  of  ap¬ 
praisal  execution: 

•  Plan  and  Prepare  for  Appraisal. 

•  Conduct  Appraisal. 

•  Report  Results. 

Requirement:  R3.  A  CAP  compliant  appraisal  method  shall  provide  guidance  for: 

•  Identifying  an  appraisal’s  goals  and  constraints. 

•  Determining  its  suitability  for  the  appraisal  in  light  of  those  goals  and 
constraints. 

Requirement:  R4.  A  CAP  compliant  appraisal  method  shall  provide  guidance  for  selecting  an 
appraisal’s  CMM  scope. 

Requirement:  R5.  A  CAP  compliant  appraisal  method  shall  provide  guidance  for  choosing  an 
appraisal’s  organizational  scope. 

Requirement:  R6.  A  CAP  compliant  appraisal  method  shall  provide  guidance  for  obtaining  an 
organization’s  commitment  to  proceed  with  the  appraisal  process. 

Requirement:  R7.  A  CAP  compliant  appraisal  method  shall  document  appraisal  team  qualifi¬ 
cation  criteria  that  conform  at  a  minimum  to  the  following  technical  and  management  experi¬ 
ence  requirements: 

•  At  least  five  years  of  technical  experience  for  the  majority  of  the  team 
members. 

•  At  least  twenty  five  years  of  technical  experience  for  the  team  as  a  whole. 

•  At  least  six  years  of  management  experience  for  one  team  member. 

•  At  least  ten  years  of  management  experience  for  the  team  as  a  whole. 

Requirement:  R8.  A  CAP  compliant  appraisal  method  shall  specify  the  qualifications  of  the 
team  leader  that  at  a  minimum  include  experience  in  using  the  appraisal  method,  managing 
teams,  facilitating  group  discussions,  and  making  presentations. 
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Requirement:  R9.  A  CAP  compliant  appraisal  method  shall  provide  guidance  for  determining 
appropriate  team  size. 

Requirement:  R10.  A  CAP  compliant  appraisal  method  shall  provide  guidance  for  preparing 
an  appraisal  team  to  conduct  an  appraisal,  including  applying  the  CMM. 

Requirement:  R1 1 .  A  CAP  compliant  appraisal  method  shall  provide  guidance  for  site  selec¬ 
tion. 

Requirement:  R12.  A  CAP  compliant  appraisal  method  shall  provide  guidance  for  project  se¬ 
lection. 

Requirement:  R13.  A  CAP  compliant  appraisal  method  shall  provide  guidance  for  participant 
selection. 

Requirement:  R14.  A  CAP  compliant  appraisal  method  shall  provide  guidance  for  appraisal 
participant  orientation. 

Requirement:  R15.  A  CAP  compliant  appraisal  method  shall  provide  guidance  for  developing 
and  documenting  an  appraisal  plan  that,  at  a  minimum: 

•  Identifies  the  appraisal  goals. 

•  Identifies  the  appraisal  scope. 

•  Identifies  the  appraisal  activities. 

•  Provides  a  schedule  for  the  activities. 

•  Identifies  the  people,  resources  and  budget  required  to  perform  the  activities. 

•  Identifies  the  appraisal  outputs  and  their  anticipated  use. 

•  Identifies  anticipated  follow-on  activities. 

•  Documents  any  planned  tailoring  of  the  appraisal  method  and  associated 
trade-offs. 

•  Identifies  risks  associated  with  appraisal  execution. 

Requirement:  R16.  A  CAP  compliant  appraisal  method  shall  provide  guidance  for  determin¬ 
ing  the  amount  of  time  required  to  conduct  an  appraisal. 

Requirement:  R17.  A  CAP  compliant  appraisal  method  shall  provide  guidance  for  appraisal 
logistics. 

Requirement:  R18.  A  CAP  compliant  appraisal  method  shall  define  a  set  of  artifacts  to  sup¬ 
port  the  following  appraisal  activities: 

•  Recording  observations. 

•  Categorizing  observations  (with  respect  to  the  CMM). 

•  Classifying  observations  (as  strengths  or  weaknesses). 

•  Validating  observations. 
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•  Recording  coverage. 

•  Making  rating  decisions. 

•  Reporting  findings  and  ratings. 

•  Managing  logistics. 

Requirement:  R19.  A  CAP  compliant  appraisal  method  shall  provide  guidance  for  implement¬ 
ing  data  collection  techniques  to  be  used  in  conducting  an  appraisal  and  classify  them  with 
respect  to  the  following  four  categories: 

•  Administering  instruments. 

•  Conducting  presentations. 

•  Conducting  interviews. 

•  Reviewing  documentation. 

Requirement:  R20.  A  CAP  compliant  appraisal  method  shall  provide  guidance  for  consolidat¬ 
ing  the  data  collected  during  an  appraisal  that  addresses  at  a  minimum: 

•  Extracting  information  from  data  gathered  during  data  collection  sessions. 

•  Recording  them  as  observations. 

•  Classifying  the  observations  (e.g.,  as  potential  strengths  or  weaknesses). 

•  Categorizing  the  observations  in  terms  of  the  CMM  or  categorizing  them  as 
non-CMM  findings. 

Requirement:  R21 .  A  CAP  compliant  appraisal  method  shall  provide  guidance  for  validating 
observations  that  includes  corroboration  of  the  observation  by  data  obtained,  at  a  minimum, 
from: 


•  Multiple  and  independent  sources. 

•  Interviews  or  presentations  that  include  people  performing  the  related  work 
or  reviews  that  include  documents  that  are  products  or  records  of  that  work. 

Requirement:  R22.  A  CAP  compliant  appraisal  method  shall  provide  guidance  for  validating 
observations  that  requires  at  a  minimum  that  a  portion  of  the  observations  related  to  each  of 
the  KPA  goals  within  the  scope  of  the  appraisal  are  supported  by  a  review  of  related  documen¬ 
tation. 

Requirement:  R23.  A  CAP  compliant  appraisal  method  shall  provide  guidance  for  evaluating 
and  documenting  the  sufficiency  of  findings  relative  to  the  scope  of  the  appraisal,  including 
coverage  criteria  that  adhere  to  the  following  minimum  set  of  rules  for  determining  whether 
sufficient  data  exists  to  support  rating: 

•  A  goal  is  covered  if  sufficient  findings  exist  to  judge  the  extent  of  its 
implementation  and  institutionalization  relative  to  the  CMM,  the  appraised 
entity,  and  the  appraised  entity’s  life  cycle(s)  (including  the  existence  of 
acceptable  alternatives). 

•  A  Key  Process  Area  is  covered  if  all  of  its  goals  are  covered. 
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•  A  maturity  level  is  covered  if  all  of  its  Key  Process  Areas  and  all  those  of 
lower  level  KPAs  are  covered. 

Requirement:  R24.  A  CAP  compliant  appraisal  method  shall  define  mechanisms  for  adjusting 
data  collection  plans  to  obtain  sufficient  coverage. 

Requirement:  R25.  A  CAP  compliant  appraisal  method  shall  provide  guidance  for  recording 
traceability  between  the  data  collected  during  the  appraisal  and  the  appraisal  outputs. 

Requirement:  R26.  A  CAP  compliant  appraisal  method  shall  require  rating  of  the  following 
categories  of  CMM  components  provided  they  fall  within  the  scope  of  the  appraisal: 

•  KPAs 

•  Goals 

Requirement:  R27.  A  CAP  compliant  appraisal  method,  If  it  calculates  a  maturity  level  rating, 
shall  require  that  the  maturity  level  rating  is  consistent  with  the  five  level  scale  provided  in  the 
CMM. 

Requirement:  R28.  A  CAP  compliant  appraisal  method  shall  define  a  rating  process  using  the 
following  rating  values: 

•  A  KPA  or  goal  is  satisfied  if  this  aspect  of  the  CMM  is  implemented  and 
institutionalized  either  as  defined  in  the  CMM,  or  with  an  adequate 
alternative. 

•  A  KPA  or  goal  is  unsatisfied  if  there  are  significant  weaknesses  in  the 
appraised  entity’s  implementation  or  institutionalization  of  this  aspect  of  the 
CMM,  as  defined,  and  no  adequate  alternative  is  in  place. 

•  A  KPA  or  goal  is  not  applicable  if  the  KPA  is  not  applicable  in  the 
organization’s  environment. 

•  A  KPA  or  goal  is  not  rated  if  the  associated  appraisal  findings  do  not  meet 
coverage  criteria  or  if  this  aspect  of  the  CMM  falls  outside  the  scope  of  the 
appraisal. 

Requirement:  R29.  A  CAP  compliant  appraisal  method  shall  define  a  rating  process  which 
specifies  that: 

•  An  appraisal  team  can  rate  a  goal  when  valid  observations  related  to  the  goal 
meet  the  method’s  defined  coverage  criteria. 

•  An  appraisal  team  can  rate  a  KPA  when  it  has  rated  each  of  the  associated 
goals. 

•  An  appraisal  team  can  determine  a  maturity  level  rating  once  it  has  rated  all 
of  the  KPAs  within  that  level  and  each  level  below. 

Requirement:  R30.  A  CAP  compliant  appraisal  method  shall  define  a  rating  process  that  re¬ 
quires  full  participation  of  all  appraisal  team  members  in  reaching  consensus  in  all  rating  de¬ 
cisions. 
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Requirement:  R31 .  A  CAP  compliant  appraisal  method  shall  require  that  ratings  be  based  on 
the  CMM  as  defined  in  Capability  Maturity  Model  for  Software,  Version  1. 1  [Paulk  93a]  and 
Key  Practices  of  the  Capability  Maturity  Model,  Version  1. 1  [Paulk  93b]. 

Requirement:  R32.  A  CAP  compliant  appraisal  method  shall  define  a  rating  process  that  re¬ 
quires  appraisal  teams  to  base  ratings  on  their  findings  (observations  that  they  have  validat¬ 
ed). 

Requirement:  R33.  A  CAP  compliant  appraisal  method  shall  define  a  rating  process  which 
specifies  that  each  goal  is  rated  in  accordance  with  the  following  rules: 

•  Rate  the  goal  “satisfied”  if  the  associated  findings  indicate  that  this  goal  is 
implemented  and  institutionalized  either  as  defined  in  the  CMM  with  no 
significant  weaknesses  or  that  an  adequate  alternative  exist. 

•  Rate  the  goal  “unsatisfied”  if  the  associated  findings  indicate  that  there  are 
significant  weaknesses  in  the  appraised  entity’s  implementation  and 
institutionalization  of  this  goal  as  defined  in  the  CMM  and  no  adequate 
alternative  is  in  place. 

•  Rate  the  goal  ‘not  applicable’  if  the  KPA  is  not  applicable  in  the  organization’s 
environment. 

•  Rate  the  goal  “not  rated”  if  the  associated  findings  do  not  meet  the  method’s 
defined  criteria  for  coverage  or  if  the  goal  falls  outside  of  the  scope  of  the 
appraisal. 

Requirement:  R34.  A  CAP  compliant  appraisal  method  shall  define  a  rating  process  which 
specifies  that  each  KPA  is  rated  in  accordance  with  the  following  rules: 

•  Rate  the  KPA  “satisfied”  if  all  of  the  goals  are  rated  “satisfied.” 

•  Rate  the  KPA  “unsatisfied”  if  any  goal  is  rated  as  “unsatisfied.” 

•  Rate  the  KPA  “not  applicable”  if  the  KPA  is  not  applicable  in  the 
organization’s  environment. 

Requirement:  R35.  A  CAP  compliant  appraisal  method  shall  define  a  rating  process  which 
specifies  that  maturity  level,  if  determined,  is  determined  in  accordance  with  the  following 
rules: 


•  A  maturity  level  is  satisfied  if  all  KPAs  within  that  level  and  each  lower  level 
are  satisfied  or  not  applicable, 

•  The  maturity  level  rating  is  that  of  the  highest  maturity  level  satisfied. 

Requirement:  R36.  A  CAP  compliant  appraisal  method  shall  identify  the  reports  that  the  ap¬ 
praisal  team  provides  as  the  result  of  the  appraisal  process. 

Requirement:  R37.  A  CAP  compliant  appraisal  method  shall  require  that  the  appraisal  team 
report  the  following  data,  at  a  minimum,  to  the  sponsor: 

•  Appraisal  scope. 

•  Appraisal  selections  (site,  projects,  participants,  team  members). 
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•  Findings. 

•  Ratings. 

•  Risks  associated  with  the  accuracy  and  completeness  of  appraisal  outputs. 

Requirement:  R38.  A  CAF  compliant  appraisal  method  shall  report  appraisal  results  to  the 
SEI. 

Requirement:  R39.  A  CAF  compliant  appraisal  method  shall  provide  guidance  for  protecting 
the  confidentiality  of  appraisal  information. 

Requirement:  R40.  A  CAF  compliant  appraisal  method  shall  provide  guidance  for  retention 
of  appraisal  records. 
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Appendix  B  Mapping  the  Key  Practices  to  Goals 


Level  2  KPAs 

Goals 

Activities 

Performed 

Commit¬ 

ment 

Abiiity 

Measure¬ 

ment 

Verifica¬ 

tion 

Requirements 

Management 

1 

1 

1 

1,2,  3,4 

1 

1,  2,  3 

2 

2,  3 

1 

3,  4 

1 

1,  2,  3 

Software  Project 
Planning 

1 

9, 10, 11, 12, 15 

1.2 

1.3,4 

1 

1,  2,  3 

2 

2,  5,  6,  7,  8, 13, 

14 

1,2 

1,  2,  3,  4 

1 

1,2,3 

3 

1,3,4 

1.2 

1,  3,4 

1 

1,  2,  3 

Software  Project 
Tracking  and  Oversight 

1 

1,5,  6,  7,  8,  9, 

10,  11,  12,  13 

1.  2 

1,  2,  3,  4,  5 

1 

1,  2,  3 

2 

2,  5,  6,  7,  8,  9, 

11 

1,2 

1,  2,  3,  4,  5 

1 

1,  2,  3 

3 

3,4 

1,2 

1,2,  3,  4,5 

1 

1,  2,  3 

Software  Subcontract 
Management 

1 

1,2 

1,2 

1,  2 

1 

2,  3 

2 

3,  4,6 

1,2 

1.  2,  3 

1 

1,2,3 

3 

7,  8,9,  13 

1.2 

1.2,3 

1 

2,  3 

4 

3,  5,  7,  9,  10, 

11, 12,  13 

1,2 

1.  2,  3 

1 

1,  2,  3 

Software  Quality 
Assurance 

1 

1,2 

1 

1,2,3 

1 

2,  3 

2 

2,  3,  4,  5 

1 

1,  2,  3,  4 

1 

2,  3 

3 

6,  7,8 

1 

1,2,  3,4 

1 

1,  2,  3 

4 

7 

1 

1,2,  3,4 

1 

1,  2,  3 

Software 

Configuration 

Management 

1 

1.2 

1 

2,  3,  4 

1 

2,4 

2 

2,  3,  4,  7 

1 

1.  2,  3,  4,  5 

1 

4 

3 

5,  6 

1 

1.2,  3,  4,5 

1 

4 

4 

8.  9,  10 

1 

2,  3,  4,  5 

1 

1,  2,  3,  4 

Table  B-1:  Level  2  KPAs:  Key  Practices  Mapped  to  Goals 


CMU/SEI-95-TR-001 


51 


Level  3  KPAs 

Goals 

Activities 

Performed 

Commit¬ 

ment 

Ability 

Measure¬ 

ment 

Verifica¬ 

tion 

Qrganization  Process 
Focus 

1 

3,  4,  5,  6,  7 

1,2,3 

1,2,  3,  4 

1 

1 

2 

1 

1.2,3 

1,  2,  3,  4 

1 

1 

3 

2 

1,2,3 

1,  2,  3,  4 

1 

1 

Qrganization  Process 
Definition 

1 

1,  2,  3,4 

1 

1,  2 

1 

1 

2 

5,  6 

1 

1,  2 

1 

1 

Training  Program 

1 

1,  2,  3 

1 

1,2,  3,4 

1 

1,  3 

2 

3,4 

1 

1,2,  3,4 

1,  2 

1,  2,  3 

3 

5,  6 

1 

1,  2,  3,  4 

1,  2 

2,  3 

Integrated  Software 
Management 

1 

1,  2,  3 

1 

1,  2 

1 

2,  3 

2 

3,  4,  5,  6,  7,  8, 

9,  10,  11 

1 

1,  3 

1 

1,  2,  3 

Software  Product 
Engineering 

1 

1,2,  3,  4,  5,  6, 

7,  8,9 

1 

1,  2,  3,  4 

1,  2 

1,  2,  3 

2 

10 

1 

1,2,  3,4 

1,  2 

1,2,3 

Intergroup 

Coordination 

1 

1 

1 

1,  2,  3,  4,  5 

1 

2,  3 

2 

3,  4,  5 

1 

1,  2,  3,  4,  5 

1 

2,  3 

3 

2,  6,7 

1 

1,2,  3,  4,5 

1 

1,  2,  3 

Peer  Reviews 

1 

1 

1 

1,2 

1 

1 

2 

2,  3 

1 

1,  2,  3 

1 

1 

Table  B-2:  Level  3  KPAs:  Key  Practices  Mapped  to  Goals 


Level  4  KPAs 


Quantitative  Process 
Management 


Software  Quality 
Management 


Goals 

Activities 

Performed 

Commit¬ 

ment 

Ability 

Measure¬ 

ment 

Verifica¬ 

tion 

1 

1,2,3 

1,2 

1.2,  4,  5 

1 

2,  3 

2 

2,  4,  5,  6 

1 

1,2,  3,  4,5 

1 

1,  2,  3 

3 

7 

2 

1,2,  3,  4,5 

1 

1,  3 

1 

1,  2 

1 

1,  2,  3 

1 

2,  3 

2 

3,  5 

1 

1,2,  3 

1 

2,  3 

3 

2,  4 

1 

1,  2,  3 

1 

1,  2,  3 

Table  B-3:  Level  4  KPAs:  Key  Practices  Mapped  to  Goals 
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Level  5  KPAs 

Goals 

Activities 

Performed 

Commit¬ 

ment 

Ability 

Measure¬ 

ment 

Verifica¬ 

tion 

Defect  Prevention 

1 

1,  2 

1,2 

1.  2,  3,  4 

1 

2,  3 

2 

3,  5 

1.2 

3,  4 

1 

3 

3 

4,  6,  7,  8 

1.2 

1,2,  3,4 

1 

1,  2,  3 

Technology  Change 
Management 

1 

1 

1,2,3 

1,2,5 

1 

2 

2 

2,  4,  5,  6 

1.2,3 

1,  2,  3,  4,  5 

1 

1,  2 

3 

3.  7,8 

1,2,3 

1,  2,  5 

1 

1,  2 

Process  Change 
Management 

1 

2,  3,4 

1.2 

1.2,3 

1 

2 

2 

1,  6,  10 

1,  2 

1,  2,  3,  4 

1 

1,  2 

3 

4,  5,  7,  8,  9 

1,2 

1,2,  3,4 

1 

1,  2 

Table  B-4:  Level  5  KPAs:  Key  Practices  Mapped  to  Goals 
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Appendix  D  Glossary 


ability  to  perform  -  One  of  five  common  features.  The  preconditions  that  must  exist  in  the 
project  or  organization  to  implement  the  software  process  competently.  Ability  to  Perform  typ¬ 
ically  involves  resources,  organization  structures,  and  training. 

accuracy  -  An  observation  is  considered  to  be  accurate  if  the  appraisal  team  agrees  that  it  is 
based  on  what  is  heard  and  seen,  is  worded  appropriately,  and  is  correctly  categorized  and 
classified. 

activities  performed-  One  of  five  common  features.  A  description  of  the  roles  and  procedures 
necessary  to  implement  a  key  process  area.  Activities  performed  typically  involves  establish¬ 
ing  plans  and  procedures,  performing  the  work,  tracking  it,  and  taking  corrective  action. 

activity- A  key  practice  of  the  activities  performed  common  feature. 

alternative  practice  -  Alternative  practices  to  those  described  in  the  CMM  that  may  accomplish 
the  goals  of  a  key  process  area. 

appraisal  constraints  -  Constraints  that  affect  appraisal  conduct  such  as  budget  limitations, 
schedule  limitations,  and  resource  limitations  (people  and  facilities). 

appraisal  goals  -  The  desired  outcome  of  an  appraisal  process. 

appraisal  requirements  -  Appraisal  constraints  and  goals. 

appraisal  scope- The  organizational  entities  and  CMM  components  selected  for  investigation. 

appraisal  sponsor  -  The  individual  who  authorizes  an  appraisal,  defines  its  goals  and  con¬ 
straints,  and  commits  to  use  of  appraisal  outputs. 

appraised  entity  -  The  organizational  units  to  which  appraisal  outputs  apply.  An  appraised  en¬ 
tity  may  be  any  portion  of  an  organization  including  an  entire  company,  a  selected  business 
unit,  a  specific  geographic  site,  units  supporting  a  particular  product  line,  units  involved  in  a 
particular  type  of  service,  an  individual  project,  or  a  multi-company  team. 

Capability  Maturity  Model  (CMM)  -  A  description  of  the  stages  through  which  software  organi¬ 
zations  evolve  as  they  define,  implement,  measure,  control,  and  improve  their  software  pro¬ 
cesses.  This  model  provides  a  guide  for  selecting  process  improvement  strategies  by 
facilitating  the  determination  of  current  process  capabilities  and  the  identification  of  the  issues 
most  critical  to  software  quality  and  process  improvement.  CMM  Version  1.1  is  specified  in 
CMU/SEI-93-TR-24  and  CMU/SEI-93-TR-25. 

categorize  observations  -  Identify  the  key  practices,  common  features,  goals,  and  KPAs  relat¬ 
ed  to  the  observation. 
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CAF-  Acronym  for  CMM  Appraisal  Framework. 

CAF  compliant  appraisal  method  -  An  appraisal  method  that  conforms  to  CAF  defined  apprais¬ 
al  method  requirements. 

CBA  -  Acronym  for  CMM-Based  Appraisal 

classify  observation  -  Classify  an  observation  as  evidence  of: 

•  A  strength  in  implementation  of  CMM  key  practices, 

•  A  weakness  in  the  implementation  of  CMM  key  practices, 

•  The  existence  of  an  alternative  practice  that  meets  KPA  goals,  or 

•  The  existence  of  a  practice  that  is  not-applicable  or  not-significant  in  an  organization's  busi¬ 
ness  context. 

CMM  -  Acronym  for  capability  maturity  model. 

CMM  Appraisal  Framework  (CAF)  -  A  framework  for  planning,  conducting,  and  completing 
CMM-based  appraisals. 

CMM-based appraisal  (CBA)  -  An  appraisal  conducted  using  a  CMM-based  appraisal  method. 

CMM-based  appraisal  method  -  An  appraisal  method  that  uses  the  CMM  as  its  framework  for 
evaluating  an  organization's  software  process. 

CMM  fidelity  -  The  use  of  CMM  components,  and  CMM  components  alone,  as  the  basis  for 
rating  an  organization's  software  process  maturity. 

CMM  scope  of  the  appraisal  -  The  portion  of  the  CMM  used  as  a  framework  for  evaluating  an 
organization's  software  process  during  an  appraisal. 

commitment-  A  pact  that  is  freely  assumed,  visible,  and  expected  to  be  kept  by  all  parties. 

commitment  to  perform  -  One  of  five  common  features.  The  actions  that  the  organization  must 
take  to  ensure  that  the  process  is  established  and  will  endure.  Commitment  to  Perform  typi¬ 
cally  involves  establishing  organizational  policies  and  senior  management  sponsorship. 

common  feature  -  The  subdivision  categories  of  the  CMM  key  process  areas.  The  common 
features  are  attributes  that  indicated  whether  the  implementation  and  institutionalization  of  a 
key  process  area  is  effective,  repeatable,  and  lasting.  The  CMM  common  features  are  the  fol¬ 
lowing: 
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•  Activities  performed, 

•  Measurement  and  analysis,  and 

•  Verifying  implementation. 

comparability  -  The  degree  to  which  the  results  of  different  appraisal  methods  are  similar. 

confidentiality-  a  characteristic  of  data  that,  by  agreement,  can  not  be  attributed  to  a  particular 
individual  or  disclosed  without  prior  agreement  or  authorization. 

consensus  -  A  method  of  decision  making  that  allows  team  members  to  develop  a  common 
basis  of  understanding  and  develop  general  agreement  concerning  a  decision. 

consistency  -  The  degree  of  uniformity,  standardization,  and  freedom  from  contradiction 
among  documents  or  system  components.  Consistency  of  an  appraisal  method  refers  to  the 
ability  of  different  appraisal  teams  using  the  same  method  to  conduct  appraisals  of  the  same 
scope  to  produce  non-conflicting  results. 

corroboration  -  Confirmation.  All  appraisal  observations  must  be  confirmed  by  information 
from  different  sources  and  different  data  gathering  sessions  prior  to  use  as  findings. 

coverage  -  The  extent  to  which  data  gathered  addresses  CMM  components,  organizational 
units,  and  life  cycle  phases  within  the  scope  of  an  appraisal. 

covered-  A  CMM  component  is  considered  to  be  covered  if  the  data  gathered  relevant  to  the 
component: 

•  Is  representative  of  the  organizational  units  within  the  scope  of  the  appraisal, 

•  Is  representative  of  the  life  cycle  phases  within  the  scope  of  the  appraisal, 

•  Addresses  an  adequate  sample  of  sub-components. 

effective  process  -  A  process  that  can  be  characterized  as  practiced,  documented,  enforced, 
trained,  measured,  and  able  to  improve. 

fact  -  A  statement  whose  content  can  be  verified  as  true  through  the  senses. 

finding  -  An  observation  that  has  been  accepted  by  the  team  as  valid.  Findings  include 
strengths,  weaknesses,  evidence  of  alternative  practices,  and  evidence  of  non-applicable 
practices.  A  set  of  findings  should  be  accurate,  corroborated,  and  consistent  within  itself. 

goal  -  A  summary  of  the  key  practices  of  a  key  process  area  that  can  be  used  to  determine 
whether  an  organization  or  project  has  effectively  implemented  the  key  process  area.  The 
goals  signify  the  scope,  boundaries,  and  intent  of  each  key  process  area. 

inference  -  A  conclusion  based  on  fact,  in  appraisals,  the  most  frequent  inferences  are  that 
the  statements  made  by  people  in  interviews  as  part  of  the  oral  discussion  are  facts. 
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judgments  -  The  exercise  of  making  sound  and  reasonable  decisions.  Judgments  refer  to  as¬ 
sertions  that  are  made  that  are  based  on  facts  and  strong  inferences. 

IIDEAL  approach  -  A  life  cycle  model  for  process  improvement.  IDEAL  stands  for  the  five 
phases  of  the  approach:  Initiating,  Diagnosing,  Establishing,  Acting,  and  Leveraging. 

institutionalization  -  The  building  of  infrastructure  and  corporate  culture  that  support  methods, 
practices,  and  procedures  so  that  they  are  the  ongoing  way  of  doing  business,  even  after 
those  who  originally  defined  them  are  gone. 

institutionalization  common  feature  -  One  of  the  four  common  features:  commitment  to  per¬ 
form,  ability  to  perform,  measurement  and  analysis,  and  verifying  implementation. 

instrument  -  Questionnaire  or  survey  used  to  collect  data  using  formal,  written  questions. 

IPI-  Acronym  for  CMM-Based  Appraisal  for  Internal  Process  Improvement. 

key  practice  -  The  infrastructures  and  activities  that  contribute  most  to  the  effective  implemen¬ 
tation  and  institutionalization  of  a  key  process  area. 

key  process  area  -  A  cluster  of  related  activities  that,  when  performed  collectively,  achieve  a 
set  of  goals  considered  important  for  establishing  process  capability.  The  key  process  areas 
have  been  identified  by  the  SEI  to  be  the  principal  building  blocks  to  help  determine  the  soft¬ 
ware  process  capability  of  an  organization  and  understand  the  improvements  needed  to  ad¬ 
vance  to  higher  maturity  levels. 

KPA  -  Acronym  for  key  process  area. 

maturity  level  -  A  well-defined  evolutionary  plateau  toward  achieving  a  mature  software  pro¬ 
cess.  The  five  maturity  levels  in  the  SEI's  Capability  Maturity  Model  are  initial,  repeatable,  de¬ 
fined,  managed,  and  optimizing. 

measurement  and  analysis  -  One  of  five  common  features.  A  description  of  the  need  to  mea¬ 
sure  the  process  and  analyze  the  measurements.  Measurement  and  Analysis  typically  in¬ 
cludes  examples  of  the  measurements  that  could  be  taken  to  determine  the  status  and 
effectiveness  of  the  Activities  Performed. 

not  applicable  -  Rating  given  to  a  CMM  component  that  is  either  not  applicable  or  insignificant 
in  an  organization's  business  environment. 

not  rated  -  Rating  given  to  a  CMM  component  that  fall  outside  the  scope  of  an  appraisal  and 
to  CMM  components  for  which  the  appraisal  team  did  not  obtain  coverage. 

observation  -  Information  extracted  from  the  notes  of  data  collection  sessions. 

organization  scope  of  an  appraisal  -  The  organizational  units  that  comprise  the  entity  being 
appraised. 
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process  -  A  sequence  of  steps  performed  for  a  given  purpose. 

process  capability  -  The  range  of  expected  results  that  can  be  achieved  by  following  a  pro¬ 
cess. 

process  maturity -The  extent  to  which  a  specific  process  is  explicitly  defined,  managed,  mea¬ 
sured,  controlled,  and  effective.  Maturity  implies  a  potential  for  growth  in  capability  and  indi¬ 
cates  both  the  richness  of  an  organization’s  software  process  and  consistency  with  which  it  is 
applied  in  projects  throughout  the  organization. 

project  -  An  undertaking  requiring  concerted  effort,  which  is  focused  on  developing  and/or 
maintaining  a  specific  product.  The  product  may  include  hardware,  software  and  other  com¬ 
ponents.  Typically  a  project  has  its  own  funding,  cost  accounting,  and  delivery  schedule. 

rating  -  A  characterization  of  an  organization's  software  process  relative  to  a  component  of  the 
CMM. 

rating  components  -  Components  of  the  CMM  that  can  be  rated  include  goals,  KPAs,  and  ma¬ 
turity  level. 

rating  scale  -  The  rating  scale  for  goals  and  KPAs  is  satisfied,  unsatisfied,  not  applicable,  and 
not  rated.  The  rating  scale  for  maturity  level  is  1  through  5. 

reliability  -  The  ability  to  attain  appraisal  results  that  accurately  characterize  an  organization's 
software  process. 

repeatability  -  The  ability  to  attain  the  same  appraisal  results  if  an  appraisal  of  identical  scope 
is  conducted  more  than  once  in  the  same  time  period. 

rules  for  corroborating  observations  -  rules  that  define  the  requirements  for  confirming  obser¬ 
vations  through  the  use  of  multiple  data  sources  and  sessions  of  prescribed  types. 

sampling- A  set  of  elements  drawn  from  and  analyzed  to  estimate  the  characteristics  of  a  pop¬ 
ulation.  During  an  appraisal  data  collection  is  planned  to  provide  a  sampling  of  the  process 
data  related  to  the  CMM  components,  organizational  units,  and  life  cycle  phases  within  the 
scope  of  the  appraisal. 

satisfied  -  Rating  given  to  a  CMM  component  that  is  applicable  in  an  organization’s  business 
environment  and  is  performed  either  as  defined  in  the  CMM  or  with  an  adequate  alternative. 

SCE-  Acronym  for  Software  Capability  Evaluation. 

SEI  -  Acronym  for  Software  Engineering  Institute. 

senior  site  manager  -  senior  manager  in  charge  of  the  appraised  entity.  The  senior  site  man¬ 
ager  may  or  may  not  be  the  appraisal  sponsor. 

site  -  a  geographic  location  of  one  or  more  of  an  organization's  units. 
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SPA  -  Acronym  for  Software  Process  Assessment. 

strength  -  Implementation  of  practices  which  in  an  appraisal  team's  judgment,  improve  an  or¬ 
ganization's  software  process  capability.  CMM  related  strengths  are  effective  implementation 
of  one  or  more  of  the  CMM  key  practices  or  one  or  more  alternative  practices  that  contribute 
equivalently  to  the  satisfaction  of  KPA  goals. 

subpractice  -  listed  beneath  top-level  key  practices  in  the  CMM  and  describe  what  one  would 
expect  to  find  implemented  for  the  top-level  key  practice.  The  subpractices  can  be  used  to  de¬ 
termine  whether  or  not  the  key  practices  are  implemented  satisfactorily. 

sufficiency  for  rating  -  The  extent  to  which  findings  meet  the  appraisal  method's  rules  for  cov¬ 
erage  and,  thus,  satisfy  the  prerequisites  for  rating. 

support  unit  -  An  organizational  level  unit  that  supports  one  or  more  projects  in  software  pro¬ 
cess  performance,  such  as  a  Software  Engineering  Process  Group. 

traceability-  The  degree  to  which  a  relationship  can  be  established  between  two  or  more  prod¬ 
ucts  of  the  appraisal  process,  especially  products  having  a  predecessor  -successor  relation¬ 
ship  to  on  another. 

unsatisfied  -  Rating  given  to  a  CMM  component  that  is  both  applicable  and  significant  in  an 
organization's  business  environment,  is  either  not  performed  or  is  performed  as  defined  in  the 
CMM  with  significant  weaknesses,  and  for  which  no  adequate  alternative  exists. 

verifying  implementation  -  One  of  five  common  features.  The  steps  to  ensure  that  the  activities 
are  performed  in  compliance  with  the  process  and  that  has  been  established.  Verification  typ¬ 
ically  encompasses  reviews  and  audits  by  management  and  software  quality  assurance. 

weakness  -  Ineffective  implementation  of  or  lack  of  practices  which  in  an  appraisal  team's 
judgment,  interfere  with  effective  performance  of  software  development  tasks.  CMM  related 
weaknesses  are  an  ineffective  implementation  or  lack  of  implementation  of  one  or  more  CMM 
key  practices  with  no  acceptable  alternative  practices  in  place. 
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